Error when I try to receive messages from Azure Service Bus - spring

I'm trying to receive messages from Azure Service Bus in my SpringBoot application, but I'm facing some issues. This is my code:
Consumer<ServiceBusReceivedMessageContext> processMessage = messageContext -> {
try {
System.out.println(messageContext.getMessage().getMessageId());
// other message processing code
messageContext.complete();
} catch (Exception ex) {
messageContext.abandon();
}
};
Consumer<ServiceBusErrorContext> processError = errorContext -> {
System.err.println("Error occurred while receiving message: " + errorContext.getException());
};
ServiceBusProcessorClient processorClient = new ServiceBusClientBuilder()
.connectionString(<<CONNECTION_STRING>>)
.processor()
.queueName("test")
.disableAutoComplete()
.receiveMode(PEEK_LOCK)
.processMessage(processMessage)
.processError(processError)
.disableAutoComplete()
.buildProcessorClient();
Basically, I'm using the same code of this example but I'm receiving this error when I run my application:
java.lang.NoSuchFieldError: CLIENT_RECEIVER_IDENTIFIER
at com.azure.messaging.servicebus.implementation.ServiceBusReactorSession.createConsumer(ServiceBusReactorSession.java:185) ~[azure-messaging-servicebus-7.11.0.jar:7.11.0]
at com.azure.messaging.servicebus.implementation.ServiceBusReactorSession.createConsumer(ServiceBusReactorSession.java:99) ~[azure-messaging-servicebus-7.11.0.jar:7.11.0]
at com.azure.messaging.servicebus.implementation.ServiceBusReactorAmqpConnection.lambda$createReceiveLink$3(ServiceBusReactorAmqpConnection.java:183) ~[azure-messaging-servicebus-7.11.0.jar:7.11.0]
at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:125) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoFlatMap$FlatMapInner.onNext(MonoFlatMap.java:249) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.complete(MonoIgnoreThen.java:292) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onNext(MonoIgnoreThen.java:187) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:236) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.onComplete(MonoIgnoreThen.java:203) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.SerializedSubscriber.onComplete(SerializedSubscriber.java:146) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.SerializedSubscriber.onComplete(SerializedSubscriber.java:146) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxTimeout$TimeoutMainSubscriber.onComplete(FluxTimeout.java:234) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoNext$NextSubscriber.onComplete(MonoNext.java:102) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:83) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxFilterFuseable$FilterFuseableSubscriber.onNext(FluxFilterFuseable.java:118) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replayNormal(FluxReplay.java:877) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replay(FluxReplay.java:965) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxReplay$ReplaySubscriber.onNext(FluxReplay.java:1344) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxPeek$PeekSubscriber.onNext(FluxPeek.java:200) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:122) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxDistinctUntilChanged$DistinctUntilChangedSubscriber.tryOnNext(FluxDistinctUntilChanged.java:149) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxDistinctUntilChanged$DistinctUntilChangedSubscriber.onNext(FluxDistinctUntilChanged.java:102) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replayNormal(FluxReplay.java:877) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.FluxReplay$SizeBoundReplayBuffer.replay(FluxReplay.java:965) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.ReplayProcessor.tryEmitNext(ReplayProcessor.java:508) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.SinkManySerialized.tryEmitNext(SinkManySerialized.java:100) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.publisher.InternalManySink.emitNext(InternalManySink.java:27) ~[reactor-core-3.4.22.jar:3.4.22]
at com.azure.core.amqp.implementation.handler.Handler.onNext(Handler.java:89) ~[azure-core-amqp-2.5.2.jar:2.5.2]
at com.azure.core.amqp.implementation.handler.SessionHandler.onSessionRemoteOpen(SessionHandler.java:87) ~[azure-core-amqp-2.5.2.jar:2.5.2]
at org.apache.qpid.proton.engine.BaseHandler.handle(BaseHandler.java:146) ~[proton-j-0.33.6.jar:na]
at org.apache.qpid.proton.engine.impl.EventImpl.dispatch(EventImpl.java:108) ~[proton-j-0.33.6.jar:na]
at org.apache.qpid.proton.reactor.impl.ReactorImpl.dispatch(ReactorImpl.java:324) ~[proton-j-0.33.6.jar:na]
at org.apache.qpid.proton.reactor.impl.ReactorImpl.process(ReactorImpl.java:291) ~[proton-j-0.33.6.jar:na]
at com.azure.core.amqp.implementation.ReactorExecutor.run(ReactorExecutor.java:91) ~[azure-core-amqp-2.5.2.jar:2.5.2]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:68) ~[reactor-core-3.4.22.jar:3.4.22]
at reactor.core.scheduler.SchedulerTask.call(SchedulerTask.java:28) ~[reactor-core-3.4.22.jar:3.4.22]
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at java.base/java.lang.Thread.run(Thread.java:834) ~[na:an]
It's weird because when I try to use the peekMessage function using the ServiceBusReceiverAsyncClient it works, but when I try to use the receiveMessages or this approach using the ProcessorClient, it throws the error. I didn't find anything about this error, I don't know if it's a property that I have to set.

Apparently it was related to te package version, previously I was using the following:
<dependency>
<groupId>com.azure</groupId>
<artifactId>azure-messaging-servicebus</artifactId>
<version>7.10.1</version>
</dependency>
when I rollback to a previous version (7.10.1), it started to work properly, I don't know if there's a compatibility issue, but now it's working properly.
Thank you.

Related

How to access Namenode UI with KnoxSSO

My goal
I want to access Namenode UI through KnoxSSO and browse hdfs files without Ambari.
What have I already done
I've already set up Security cluster and HDFS HA with QJM.
Apache Knox uses Pac4J provider to authenticate users with Openid Connect.
There are 3 instances of Apache Knox beyond Nginx reverse proxy.
I've also have several services working fine with KnoxSSO:
Resourcemanager UI
Jobhistory UI
WebHDFS
Apache Zeppelin
Spark History Server
The problem
Despite of all other services are accessable by their UI through knox, I can't achieve it with Namenode IU (HDFSUI).
I folowed instructions by official documentation but when I try to follow hdfsui link it's cause probably infinite loop redirecting and show: HTTP ERROR 500 java.io.IOException: Service connectivity error.
In the gateway.log file I have numerous exceptions, like these:
2022-03-19 18:15:35,509 WARN knox.gateway (IdentityAsserterHttpServletRequestWrapper.java:scrubOfExistingPrincipalParams(202)) - Possible identity spoofing attempt - impersonation parameter removed: doAs
2022-03-19 18:15:35,521 WARN knox.gateway (DefaultDispatch.java:executeOutboundRequest(183)) - Connection exception dispatching request: https://my.hadoop.domain:443/gateway/default/hdfs?doAs=username java.net.SocketTimeoutException: Read timed out
java.net.SocketTimeoutException: Read timed out
at java.base/java.net.SocketInputStream.socketRead0(Native Method)
at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:472)
at java.base/sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(SSLSocketInputRecord.java:70)
at java.base/sun.security.ssl.SSLSocketImpl.readApplicationRecord(SSLSocketImpl.java:1374)
at java.base/sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:985)
at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:166)
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:152)
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequestWrapper(DefaultDispatch.java:135)
at org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:300)
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:174)
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:125)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193)
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:149)
at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:94)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:57)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter$1.run(AbstractJWTFilter.java:249)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at org.apache.knox.gateway.provider.federation.jwt.filter.AbstractJWTFilter.continueWithEstablishedSecurityContext(AbstractJWTFilter.java:244)
at org.apache.knox.gateway.provider.federation.jwt.filter.SSOCookieFederationFilter.doFilter(SSOCookieFederationFilter.java:164)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:50)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:166)
at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:93)
at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:135)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:41)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:106)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.websocket.server.WebSocketHandler.handle(WebSocketHandler.java:115)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-03-19 18:15:35,522 ERROR knox.gateway (AbstractGatewayFilter.java:doFilter(60)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2022-03-19 18:15:35,522 ERROR knox.gateway (AbstractGatewayFilter.java:doFilter(60)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2022-03-19 18:15:35,522 ERROR knox.gateway (AbstractGatewayFilter.java:doFilter(60)) - Failed to execute filter: java.io.IOException: Service connectivity error.
2022-03-19 18:15:35,523 ERROR knox.gateway (GatewayFilter.java:doFilter(168)) - Gateway processing failed: java.io.IOException: Service connectivity error.
java.io.IOException: Service connectivity error.
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeOutboundRequest(DefaultDispatch.java:184)
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequest(DefaultDispatch.java:152)
at org.apache.knox.gateway.dispatch.DefaultDispatch.executeRequestWrapper(DefaultDispatch.java:135)
at org.apache.knox.gateway.dispatch.DefaultDispatch.doGet(DefaultDispatch.java:300)
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter$GetAdapter.doMethod(GatewayDispatchFilter.java:174)
at org.apache.knox.gateway.dispatch.GatewayDispatchFilter.doFilter(GatewayDispatchFilter.java:125)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.doFilterInternal(AbstractIdentityAssertionFilter.java:193)
at org.apache.knox.gateway.identityasserter.common.filter.AbstractIdentityAssertionFilter.continueChainAsPrincipal(AbstractIdentityAssertionFilter.java:149)
at org.apache.knox.gateway.identityasserter.common.filter.CommonIdentityAssertionFilter.doFilter(CommonIdentityAssertionFilter.java:94)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.filter.rewrite.api.UrlRewriteServletFilter.doFilter(UrlRewriteServletFilter.java:57)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.filter.XForwardedHeaderFilter.doFilter(XForwardedHeaderFilter.java:50)
at org.apache.knox.gateway.filter.AbstractGatewayFilter.doFilter(AbstractGatewayFilter.java:58)
at org.apache.knox.gateway.GatewayFilter$Holder.doFilter(GatewayFilter.java:348)
at org.apache.knox.gateway.GatewayFilter$Chain.doFilter(GatewayFilter.java:262)
at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:166)
at org.apache.knox.gateway.GatewayFilter.doFilter(GatewayFilter.java:93)
at org.apache.knox.gateway.GatewayServlet.service(GatewayServlet.java:135)
at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791)
at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:228)
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1612)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1582)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.trace.TraceHandler.handle(TraceHandler.java:51)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.filter.CorrelationHandler.handle(CorrelationHandler.java:41)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.apache.knox.gateway.filter.PortMappingHelperHandler.handle(PortMappingHelperHandler.java:106)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.websocket.server.WebSocketHandler.handle(WebSocketHandler.java:115)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:540)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:395)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905)
at java.base/java.lang.Thread.run(Thread.java:829)
In debug logs I saw some strange headers like that
X-Forwarded-Context: /gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default/gateway/default
Could anyone know, what's going wrong?
Try to add the following line for HDFSUI role in your topology file:
<version>2.7.0</version>
It should be look like this:
<service>
<role>HDFSUI</role>
<version>2.7.0</version>
<url>http://NAMENODE:50070</url>
</service>
https://issues.apache.org/jira/browse/KNOX-1644

Spring cloud gateway - io.netty.util.internal.OutOfDirectMemoryError when using modifyRequestBody filter

I am using
Spring Cloud Gateway version 3.0.3
reactor-netty-core 1.0.10
and i am facing a io.netty.util.internal.OutOfDirectMemoryError when the spring cloud gateway modifyRequestBody filter is being used. When it is being commented out, load tests runs fine without any error.
In our filters, we are doing a cache of the request body which will be used in our Fallback when there is a timeout or errors. Cache is just using the default ServerWebExchangeUtils.cacheRequestBody method.Fallback is implemented using Resilience4j Circuit Breaker.
In part of our chain, we are also using the spring gateway modifyRequestBody filter to modify our request body before sending it to the resource server.
OUT 2022-01-19 18:10:54.344 ERROR 16 --- [r-http-epoll-12] io.netty.util.ResourceLeakDetector : LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
OUT Recent access records:
OUT Created at:
OUT io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
OUT io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
OUT io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:115)
OUT org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:71)
OUT org.springframework.core.io.buffer.NettyDataBufferFactory.allocateBuffer(NettyDataBufferFactory.java:39)
OUT org.springframework.core.codec.CharSequenceEncoder.encodeValue(CharSequenceEncoder.java:91)
OUT org.springframework.core.codec.CharSequenceEncoder.lambda$encode$0(CharSequenceEncoder.java:75)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:106)
OUT reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:151)
OUT reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
OUT reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:127)
OUT reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
OUT reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1817)
OUT reactor.core.publisher.FluxCallable.subscribe(FluxCallable.java:49)
OUT reactor.core.publisher.Mono.subscribe(Mono.java:4338)
OUT reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
OUT reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
OUT reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.Mono.subscribe(Mono.java:4338)
OUT reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:255)
OUT reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:64)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:52)
OUT reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:157)
OUT reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onNext(FluxMap.java:120)
OUT reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:101)
OUT reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
OUT reactor.core.publisher.FluxMapFuseable$MapFuseableConditionalSubscriber.onNext(FluxMapFuseable.java:295)
OUT reactor.core.publisher.FluxFilterFuseable$FilterFuseableConditionalSubscriber.onNext(FluxFilterFuseable.java:337)
OUT reactor.core.publisher.Operators$MonoSubscriber.complete(Operators.java:1816)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onComplete(MonoCollect.java:159)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:142)
OUT reactor.netty.channel.FluxReceive.terminateReceiver(FluxReceive.java:468)
OUT reactor.netty.channel.FluxReceive.drainReceiver(FluxReceive.java:260)
OUT reactor.netty.channel.FluxReceive.request(FluxReceive.java:129)
OUT reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:162)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.request(FluxPeek.java:138)
OUT reactor.core.publisher.FluxMap$MapSubscriber.request(FluxMap.java:162)
OUT reactor.core.publisher.MonoCollect$CollectSubscriber.onSubscribe(MonoCollect.java:103)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
OUT reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:171)
OUT reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
OUT reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:167)
OUT reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:146)
OUT io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
OUT io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
OUT io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384)
OUT io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
OUT io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
OUT io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
OUT java.lang.Thread.run(Thread.java:748)
Netty.docs: Reference counted objects
Sample Code Snippet:
return predicateSpec
.path(pathPattern)
.filters(f -> f
.filter(cachingRequestBodyFilter)
.modifyRequestBody(String.class, String.class, (exchange, modified) -> {
//Modify request body method here
})
.circuitBreaker(c -> c.setName("sessionTimeoutCircuitBreaker").setFallbackUri("forward:/timeoutFallback/"+country+"/"+domain))
)
.uri(uri);

Spring Cloud Dataflow Duplicate key error (A job execution for this job is already running)

I have a Spring Cloud Dataflow Stream with an Http source, custom processor and Task-Launcher
I have a composed task that gets called from the stream - task-launcher sink
The composed task gets launched correctly and runs successfully when only 1 instance is executed but when multiple instances are created they intermittently have errors with duplicate keys. I have recently updated to the following versions:
• bitnami/spring-cloud-dataflow
version: 2.9.0-debian-10-r15
• bitnami/spring-cloud-dataflow-composed-task-runner
version: 2.9.0-debian-10-r17
• bitnami/spring-cloud-skipper
version: 2.8.0-debian-10-r15
java.lang.IllegalStateException: Failed to execute ApplicationRunner
2021-10-28T16:53:59.179909948Z at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:785) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179913386Z at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:772) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179916469Z at org.springframework.boot.SpringApplication.run(SpringApplication.java:345) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179919669Z at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179922662Z at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332) [spring-boot-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.179925769Z at org.springframework.cloud.dataflow.composedtaskrunner.ComposedTaskRunner.main(ComposedTaskRunner.java:31) [classes!/:2.9.0]
2021-10-28T16:53:59.179928817Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.179931884Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.179934872Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.179937847Z at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.179940922Z at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179944118Z at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179947158Z at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179950209Z at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) [spring-cloud-dataflow-composed-task-runner-2.9.0.jar:2.9.0]
2021-10-28T16:53:59.179953277Z Caused by: org.springframework.batch.core.repository.JobExecutionAlreadyRunningException: A job execution for this job is already running: JobInstance: id=1200, version=0, Job=[composedtask]
2021-10-28T16:53:59.179957212Z at org.springframework.batch.core.repository.support.SimpleJobRepository.createJobExecution(SimpleJobRepository.java:127) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.179960544Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.179963557Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.179975310Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.179978588Z at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.179981557Z at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179984657Z at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179987854Z at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179990997Z at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179994173Z at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.179997313Z at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180000409Z at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180003545Z at org.springframework.batch.core.repository.support.AbstractJobRepositoryFactoryBean$1.invoke(AbstractJobRepositoryFactoryBean.java:181) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180006834Z at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180009937Z at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180013021Z at com.sun.proxy.$Proxy64.createJobExecution(Unknown Source) ~[na:na]
2021-10-28T16:53:59.180016256Z at org.springframework.batch.core.launch.support.SimpleJobLauncher.run(SimpleJobLauncher.java:137) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180019375Z at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_312]
2021-10-28T16:53:59.180022367Z at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_312]
2021-10-28T16:53:59.180025349Z at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_312]
2021-10-28T16:53:59.180028473Z at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_312]
2021-10-28T16:53:59.180031405Z at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180034563Z at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180037663Z at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180042900Z at org.springframework.batch.core.configuration.annotation.SimpleBatchConfiguration$PassthruAdvice.invoke(SimpleBatchConfiguration.java:128) ~[spring-batch-core-4.3.3.jar!/:4.3.3]
2021-10-28T16:53:59.180046230Z at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180049362Z at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) ~[spring-aop-5.3.10.jar!/:5.3.10]
2021-10-28T16:53:59.180052538Z at com.sun.proxy.$Proxy67.run(Unknown Source) ~[na:na]
2021-10-28T16:53:59.180055536Z at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.execute(JobLauncherApplicationRunner.java:199) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180058822Z at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.executeLocalJobs(JobLauncherApplicationRunner.java:173) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180061980Z at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.launchJobFromProperties(JobLauncherApplicationRunner.java:160) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180065453Z at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:155) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180068728Z at org.springframework.boot.autoconfigure.batch.JobLauncherApplicationRunner.run(JobLauncherApplicationRunner.java:150) ~[spring-boot-autoconfigure-2.5.5.jar!/:2.5.5]
2021-10-28T16:53:59.180071912Z at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:782) [spring-boot-2.5.5.jar!/:2.5.5]
I reported this issue a few months back when we were running scdf 2.8.1, I was told that the issue had been acknowledged and would be corrected in a future release. Any updates on this issue or workarounds?
I have tried using the --increment-instance-enabled=true on the task-launcher sink but this did not help.
Thanks!
Unfortunately, it appears, we didn't backport the bug-fix (ie: 4684) to the 2.8.x branch, so that's why it is not yet available.
That said, though, this fix is included in the latest 2.9.0/2.9.1 releases — please give it a go and let us know if you see an anomaly.
After upgrading Spring Cloud Dataflow to 2.9.0
I Finally figured this out, with some help from VMWare. The answer is to set the uuid-instance-enabled property=true.
Initially I was trying to set this on the stream deployment properties for the task-launcher and this was incorrect.
In the custom processor where the task-launch request message is created, I set the uuid-instance-enabled=true property and this corrected the issue. The properties set here will filter down to the composed task runner that is spun up (kubernetes pod) to execute each composed task.
Example code:
List<String> composedTaskArgs = new ArrayList<>();
composedTaskArgs.add("--dataflow-server-uri="+scdfProperties.getDataflowServerUri());
composedTaskArgs.add("--uuid-instance-enabled=true");
json.put("args", composedTaskArgs);
json.put("deploymentProps", props);
message = json.toString();
logger.info("Send to tasklauncher: "+message);
return new GenericMessage<>(message);

WSO2 issue creating organization using TenantMgtAdminServiceStub

I wrote a code that adds a tenant to WSO2 using TenantMgtAdminServiceStub.
The tenant is added successfully, however when I try to add API to it using APIM 1.10.0 CreateApi::postApis, I get an exception (all details will follow below).
Only after I login to the admin of the newly created tenant's in the publisher UI (or by curl to login.jag) I'm able to add my API using the same CreateApi::postApis.
Here are the details.
The code to add the tenant:
System.setProperty("javax.net.ssl.trustStore", clientTrustStore.getAbsolutePath());
System.setProperty("javax.net.ssl.trustStorePassword", "admin");
System.setProperty("javax.net.ssl.trustStoreType", "admin");
System.setProperty("carbon.repo.write.mode", "true");
TenantMgtAdminServiceStub stub = new TenantMgtAdminServiceStub("https://localhost:9443/services/TenantMgtAdminService.TenantMgtAdminServiceHttpsSoap12Endpoint/");
ServiceClient sc = stub._getServiceClient();
setBasicAccessSecurityHeaders(adminUser, adminPasswd, sc);
try
{
TenantInfoBean tenant = new TenantInfoBean();
tenant.setActive(true);
tenant.setAdmin("admin");
tenant.setAdminPassword("admin");
tenant.setTenantDomain("testing123.com");
tenant.setFirstname("first");
tenant.setLastname("last");
tenant.setEmail("email#testing123.com");
stub.addTenant(tenant);
}
catch (RemoteException e) { ... }
catch (TenantMgtAdminServiceExceptionException e) { ... }
finally
{
stub.cleanup();
}
The code that adds the API:
ApiClient apiClient = Configuration.getDefaultApiClient();
apiClient.setBasePath("https://localhost:9443/api/am/publisher/v0.9");
apiClient.getAuthentications().put("BasicAuth", new HttpBasicAuth());
apiClient.setUsername("admin#testing123.com");
apiClient.setPassword("admin");
apiClient.setDebugging(true);
CreateApi crApi = new CreateApi(apiClient);
API body = new API();
... invoke setters of body ...
API createdApi = crApi.apisPost(body, "application/json");
...
Doing so I receive the following exception:
TID: [11] [] [2016-04-13 10:13:37,065] admin#testing123.com#testing123.com [11] [AM]ERROR {org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl} - Error in associating lifecycle for the artifact. id: b299a2ca-ae27-4198-a8dc-f69afda05379, path: /apimgt/applicationdata/provider/admin-AT-testing123.com/partnerInfo/1.0.0/api. {org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl}
org.wso2.carbon.registry.core.exceptions.RegistryException: Couldn't find aspectName 'APILifeCycle'
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.associateAspect(EmbeddedRegistry.java:2428)
at org.wso2.carbon.registry.core.caching.CacheBackedRegistry.associateAspect(CacheBackedRegistry.java:458)
at org.wso2.carbon.registry.core.session.UserRegistry.associateAspectInternal(UserRegistry.java:1900)
at org.wso2.carbon.registry.core.session.UserRegistry.access$3800(UserRegistry.java:60)
at org.wso2.carbon.registry.core.session.UserRegistry$39.run(UserRegistry.java:1872)
at org.wso2.carbon.registry.core.session.UserRegistry$39.run(UserRegistry.java:1869)
at java.security.AccessController.doPrivileged(Native Method)
at org.wso2.carbon.registry.core.session.UserRegistry.associateAspect(UserRegistry.java:1869)
at org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl.attachLifecycle(GovernanceArtifactImpl.java:378)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.createAPI(APIProviderImpl.java:2119)
at org.wso2.carbon.apimgt.impl.APIProviderImpl.addAPI(APIProviderImpl.java:574)
at org.wso2.carbon.apimgt.impl.UserAwareAPIProvider.addAPI(UserAwareAPIProvider.java:48)
at org.wso2.carbon.apimgt.rest.api.publisher.impl.ApisApiServiceImpl.apisPost(ApisApiServiceImpl.java:180)
at org.wso2.carbon.apimgt.rest.api.publisher.ApisApi.apisPost(ApisApi.java:68)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:180)
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:96)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:200)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:99)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:59)
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:96)
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:308)
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:251)
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:234)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:208)
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:160)
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:180)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:293)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:212)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:268)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:57)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:62)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:159)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:57)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1739)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1698)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Now, as a workaround, I simply log into the admin of this newly created organization:
curl -X POST -c cookies http://localhost:9763/publisher/site/blocks/user/login/ajax/login.jag -d 'action=login&username=admin#testing123.com&password=admin'
And now when I'm trying to add API using the same code exactly, the API is added successfully.
I believe the meaningful part of the exception is:
{org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl} -
Error in associating lifecycle for the artifact.
id: b299a2ca-ae27-4198-a8dc-f69afda05379,
path: /apimgt/applicationdata/provider/admin-AT-testing123.com/partnerInfo/1.0.0/api.
{org.wso2.carbon.governance.api.common.dataobjects.GovernanceArtifactImpl}
org.wso2.carbon.registry.core.exceptions.RegistryException:
Couldn't find aspectName 'APILifeCycle'
at org.wso2.carbon.registry.core.jdbc.EmbeddedRegistry.associateAspect(EmbeddedRegistry.java:2428)
Again, I did not change anything in the code to make the postApis work, only logged-in to the admin user and it resolved by itself by re-run of the same code.
I cannot tell what exactly is the root cause.

How to reconnect to a persistence unit after an oracle Database failure

When I unfortunately need to restart my Oracle database (after a failure, or for any reason). My persistence unit cannot reconnect to the database (after it is up again).
Here is the error log :
[#|2012-03-28T16:59:01.384+0200|WARNING|glassfish3.1.1|org.eclipse.persistence.session.file:/home/distriapps/glassfish3/glassfish/domains/distriapps/applications/SiteApplicationUpdater-Central/SAU-CommonFacade-ejb_jar/_SiteApplicationUpdater_PU|_ThreadID=40;_ThreadName=Thread-2;|
Local Exception Stack:
Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.3.0.v20110604-r9504): org.eclipse.persistence.exceptions.DatabaseException
Internal Exception: java.sql.SQLRecoverableException: Closed Connection
Error Code: 17008
Call: SELECT t0.ID, t0.ADDRESS, t0.CITY, t0.COUNTRY, t0.DESCRIPTION, t0.EXHIBITORNAME, t0.GLASSFISHVERSION, t0.LASTSYNCTIMESTAMP, t0.LASTUPDATETIMESTAMP, t0.MACHINENAME, t0.OSARCH, t0.OSNAME, t0.OSVERSION, t0.SITEAPPLICATIONMACHINEID, t0.ZIPCODE FROM SAU_REPORTS t0 WHERE ((SELECT COUNT(t1.ID) FROM SAU_ISLINKS t1 WHERE (t1.SiteApplicationReport_id = t0.ID)) = ?)
bind => [1 parameter bound]
Query: ReadAllQuery(referenceClass=SiteApplicationReport sql="SELECT t0.ID, t0.ADDRESS, t0.CITY, t0.COUNTRY, t0.DESCRIPTION, t0.EXHIBITORNAME, t0.GLASSFISHVERSION, t0.LASTSYNCTIMESTAMP, t0.LASTUPDATETIMESTAMP, t0.MACHINENAME, t0.OSARCH, t0.OSNAME, t0.OSVERSION, t0.SITEAPPLICATIONMACHINEID, t0.ZIPCODE FROM SAU_REPORTS t0 WHERE ((SELECT COUNT(t1.ID) FROM SAU_ISLINKS t1 WHERE (t1.SiteApplicationReport_id = t0.ID)) = ?)")
at org.eclipse.persistence.exceptions.DatabaseException.sqlException(DatabaseException.java:333)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.processExceptionForCommError(DatabaseAccessor.java:1494)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:636)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.executeCall(DatabaseAccessor.java:535)
at org.eclipse.persistence.internal.sessions.AbstractSession.basicExecuteCall(AbstractSession.java:1702)
at org.eclipse.persistence.sessions.server.ServerSession.executeCall(ServerSession.java:566)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:207)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeCall(DatasourceCallQueryMechanism.java:193)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.executeSelectCall(DatasourceCallQueryMechanism.java:264)
at org.eclipse.persistence.internal.queries.DatasourceCallQueryMechanism.selectAllRows(DatasourceCallQueryMechanism.java:646)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRowsFromTable(ExpressionQueryMechanism.java:2592)
at org.eclipse.persistence.internal.queries.ExpressionQueryMechanism.selectAllRows(ExpressionQueryMechanism.java:2551)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:418)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1097)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:829)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1056)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:390)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1144)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2863)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1501)
at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1571)
at org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5453)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1537)
at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1571)
at org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5453)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1537)
at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1571)
at org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5453)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1537)
at org.eclipse.persistence.internal.sessions.AbstractSession.retryQuery(AbstractSession.java:1571)
at org.eclipse.persistence.sessions.server.ClientSession.retryQuery(ClientSession.java:593)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.retryQuery(UnitOfWorkImpl.java:5453)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1537)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1483)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1457)
at org.eclipse.persistence.internal.jpa.EJBQueryImpl.executeReadQuery(EJBQueryImpl.java:485)
at org.eclipse.persistence.internal.jpa.EJBQueryImpl.getResultList(EJBQueryImpl.java:742)
at com.xdcinema.siteapplicationupdater.commonfacade.SiteApplicationReportFacade.findUnlinkedSite(SiteApplicationReportFacade.java:275)
at sun.reflect.GeneratedMethodAccessor2440.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:1052)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:1124)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:5366)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:619)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:571)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doAround(SystemInterceptorProxy.java:162)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:144)
at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:861)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:800)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:370)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:5338)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:5326)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:214)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:88)
at $Proxy376.findUnlinkedSite(Unknown Source)
at com.xdcinema.siteapplicationupdater.centralserver.jsf.SiteApplicationReportController.getItems(SiteApplicationReportController.java:598)
at sun.reflect.GeneratedMethodAccessor366.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at javax.el.BeanELResolver.getValue(BeanELResolver.java:302)
at com.sun.faces.el.DemuxCompositeELResolver._getValue(DemuxCompositeELResolver.java:176)
at com.sun.faces.el.DemuxCompositeELResolver.getValue(DemuxCompositeELResolver.java:203)
at com.sun.el.parser.AstValue.getValue(AstValue.java:116)
at com.sun.el.parser.AstValue.getValue(AstValue.java:163)
at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:219)
at com.sun.faces.facelets.el.TagValueExpression.getValue(TagValueExpression.java:109)
at javax.faces.component.ComponentStateHelper.eval(ComponentStateHelper.java:194)
at javax.faces.component.ComponentStateHelper.eval(ComponentStateHelper.java:182)
at javax.faces.component.UIData.getValue(UIData.java:731)
at org.primefaces.component.datatable.DataTable.getValue(DataTable.java:740)
at javax.faces.component.UIData.getDataModel(UIData.java:1798)
at javax.faces.component.UIData.getRowCount(UIData.java:356)
at org.primefaces.component.datatable.DataTableRenderer.encodeTbody(DataTableRenderer.java:600)
at org.primefaces.component.datatable.DataTableRenderer.encodeEnd(DataTableRenderer.java:91)
at javax.faces.component.UIComponentBase.encodeEnd(UIComponentBase.java:875)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1763)
at com.sun.faces.context.PartialViewContextImpl$PhaseAwareVisitCallback.visit(PartialViewContextImpl.java:518)
at com.sun.faces.component.visit.PartialVisitContext.invokeVisitCallback(PartialVisitContext.java:183)
at javax.faces.component.UIData.visitTree(UIData.java:1411)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1600)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1600)
at javax.faces.component.UIForm.visitTree(UIForm.java:344)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1600)
at javax.faces.component.UIComponent.visitTree(UIComponent.java:1600)
at com.sun.faces.context.PartialViewContextImpl.processComponents(PartialViewContextImpl.java:376)
at com.sun.faces.context.PartialViewContextImpl.processPartial(PartialViewContextImpl.java:297)
at javax.faces.context.PartialViewContextWrapper.processPartial(PartialViewContextWrapper.java:183)
at javax.faces.component.UIViewRoot.encodeChildren(UIViewRoot.java:981)
at javax.faces.component.UIComponent.encodeAll(UIComponent.java:1756)
at com.sun.faces.application.view.FaceletViewHandlingStrategy.renderView(FaceletViewHandlingStrategy.java:390)
at com.sun.faces.application.view.MultiViewHandler.renderView(MultiViewHandler.java:131)
at com.sun.faces.lifecycle.RenderResponsePhase.execute(RenderResponsePhase.java:121)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:101)
at com.sun.faces.lifecycle.LifecycleImpl.render(LifecycleImpl.java:139)
at javax.faces.webapp.FacesServlet.service(FacesServlet.java:594)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1539)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:281)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:655)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:595)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:98)
at com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:91)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:162)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:330)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:231)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:232)
at com.sun.grizzly.http.ProcessorTask.invokeAdapter(ProcessorTask.java:828)
at com.sun.grizzly.http.ProcessorTask.doProcess(ProcessorTask.java:725)
at com.sun.grizzly.http.ProcessorTask.process(ProcessorTask.java:1019)
at com.sun.grizzly.http.DefaultProtocolFilter.execute(DefaultProtocolFilter.java:225)
at com.sun.grizzly.DefaultProtocolChain.executeProtocolFilter(DefaultProtocolChain.java:137)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:104)
at com.sun.grizzly.DefaultProtocolChain.execute(DefaultProtocolChain.java:90)
at com.sun.grizzly.http.HttpProtocolChain.execute(HttpProtocolChain.java:79)
at com.sun.grizzly.ProtocolChainContextTask.doCall(ProtocolChainContextTask.java:54)
at com.sun.grizzly.SelectionKeyContextTask.call(SelectionKeyContextTask.java:59)
at com.sun.grizzly.ContextTask.run(ContextTask.java:71)
at com.sun.grizzly.util.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:532)
at com.sun.grizzly.util.AbstractThreadPool$Worker.run(AbstractThreadPool.java:513)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.sql.SQLRecoverableException: Closed Connection
at oracle.jdbc.driver.PhysicalConnection.prepareStatement(PhysicalConnection.java:3560)
at com.sun.gjc.spi.base.ConnectionHolder.prepareStatement(ConnectionHolder.java:562)
at com.sun.gjc.spi.jdbc40.ConnectionWrapper40.prepareCachedStatement(ConnectionWrapper40.java:255)
at com.sun.gjc.spi.jdbc40.ConnectionWrapper40.prepareCachedStatement(ConnectionWrapper40.java:52)
at com.sun.gjc.spi.ManagedConnection.prepareCachedStatement(ManagedConnection.java:964)
at com.sun.gjc.spi.jdbc40.ConnectionWrapper40.prepareStatement(ConnectionWrapper40.java:173)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1474)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.prepareStatement(DatabaseAccessor.java:1423)
at org.eclipse.persistence.internal.databaseaccess.DatabaseCall.prepareStatement(DatabaseCall.java:697)
at org.eclipse.persistence.internal.databaseaccess.DatabaseAccessor.basicExecuteCall(DatabaseAccessor.java:585)
... 125 more|#]
EclipseLink is used in a JDBC connection Pool with Glassfish 3.1.1 .
Do you know a convenient method for the pool to reconnect automatically after losing the connection to the database ?
I think that the problem is not caused by EclipseLink but by the glassfish connection pool which you use.
The pool seems to be not aware that its connections are closed/invalid and still returning them to your application/eclipselink.
I never configured this pool, but please consult the documentation how to enable the keep alive/ verification feature within the connection pool.
This will ensure that the pool will test idle connections periodically and remove/renew them in case they are detected as invalid/closed.

Resources