I'm running into a problem with Quarkus Reactive when running load tests. It seems connections aren't freed up fast enough.
The endpoint basically does a database lookup by a received code, and then stores a new event with the entity as a relation.
#GET
#PermitAll
#Path("s/")
#Produces(MediaType.TEXT_PLAIN)
public Uni<Response> request(#PathParam("code") String code) {
return myService.fetchEntityByCode(code)
.onItem().invoke(entity-> {
//persist the events here
persist(SomeEvent.builder().entity(entity).build());
})
.onItem().transform(entity-> {
Response.ResponseBuilder builder = new ResponseBuilderImpl();
//do some other things to build the response
return builder.build();
});
private void persist(MyEvent myEvent) {
Panache.withTransaction(myEvent::persist)
.onFailure().invoke(throwable -> log.error("Persisting failed", throwable))
.subscribe().with(p -> {
//just a dummy subscriber to get "fire&forget" behaviour
});
}
When I run the load test with k6, I'm getting the error below. I'm assuming this is related because connections are not freed up again, and then after a while there are problems creating new connections to the database.
2022-01-30 17:04:04,865 DEBUG [org.hib.res.jdb.int.LogicalConnectionManagedImpl] (vert.x-eventloop-thread-6) `hibernate.connection.provider_disables_autocommit` was enabled. This setting should only be enabled when you are certain that the Connections given to Hibernate by the ConnectionProvider have auto-commit disabled. Enabling this setting when the Connections do not have auto-commit disabled will lead to Hibernate executing SQL operations outside of any JDBC/SQL transaction.
2022-01-30 17:04:05,052 ERROR [com.lau.lin.MyResource] (vert.x-eventloop-thread-11) Persisting failed
2022-01-30 17:04:05,052 ERROR [io.qua.mut.run.MutinyInfrastructure] (vert.x-eventloop-thread-11) Mutiny had to drop the following exception: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
[Exception 0] io.vertx.core.impl.NoStackTraceThrowable: Timeout
[Exception 1] java.lang.IllegalStateException: HR000060: Session is closed
at io.smallrye.mutiny.groups.UniOnFailure.lambda$call$3(UniOnFailure.java:108)
at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform$UniOnFailureTransformProcessor.onFailure(UniOnFailureTransform.java:54)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:48)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forwardResult(UniCreateFromCompletionStage.java:60)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:144)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forward(UniCreateFromCompletionStage.java:51)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:35)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:101)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:83)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:48)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:48)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:48)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forwardResult(UniCreateFromCompletionStage.java:58)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)
at io.vertx.core.Future.lambda$toCompletionStage$2(Future.java:362)
at io.vertx.core.impl.future.FutureImpl$3.onFailure(FutureImpl.java:153)
at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)
at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
at io.vertx.core.impl.future.Mapping.onFailure(Mapping.java:45)
at io.vertx.core.impl.future.FutureBase.emitFailure(FutureBase.java:75)
at io.vertx.core.impl.future.FutureImpl.tryFail(FutureImpl.java:230)
at io.vertx.core.impl.future.PromiseImpl.tryFail(PromiseImpl.java:23)
at io.vertx.core.impl.future.PromiseImpl.onFailure(PromiseImpl.java:54)
at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:43)
at io.vertx.core.impl.future.PromiseImpl.handle(PromiseImpl.java:23)
at io.vertx.sqlclient.impl.pool.SqlConnectionPool$1PoolRequest.lambda$null$0(SqlConnectionPool.java:202)
at io.vertx.core.net.impl.pool.SimpleConnectionPool$Cancel.run(SimpleConnectionPool.java:666)
at io.vertx.core.net.impl.pool.CombinerExecutor.submit(CombinerExecutor.java:50)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.execute(SimpleConnectionPool.java:240)
at io.vertx.core.net.impl.pool.SimpleConnectionPool.cancel(SimpleConnectionPool.java:629)
at io.vertx.sqlclient.impl.pool.SqlConnectionPool$1PoolRequest.lambda$onEnqueue$1(SqlConnectionPool.java:199)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:893)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:860)
at io.vertx.core.impl.EventLoopContext.emit(EventLoopContext.java:50)
at io.vertx.core.impl.DuplicatedContext.emit(DuplicatedContext.java:168)
at io.vertx.core.impl.AbstractContext.emit(AbstractContext.java:53)
at io.vertx.core.impl.VertxImpl$InternalTimerHandler.run(VertxImpl.java:883)
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:503)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Suppressed: java.lang.IllegalStateException: HR000060: Session is closed
at org.hibernate.reactive.pool.impl.ProxyConnection.withConnection(ProxyConnection.java:51)
at org.hibernate.reactive.pool.impl.ProxyConnection.rollbackTransaction(ProxyConnection.java:154)
at org.hibernate.reactive.mutiny.impl.MutinySessionImpl$Transaction.rollback(MutinySessionImpl.java:459)
at org.hibernate.reactive.mutiny.impl.MutinySessionImpl$Transaction.lambda$execute$3(MutinySessionImpl.java:441)
at io.smallrye.context.impl.wrappers.SlowContextualSupplier.get(SlowContextualSupplier.java:21)
at io.smallrye.mutiny.groups.UniOnFailure.lambda$call$5(UniOnFailure.java:133)
at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
at io.smallrye.mutiny.groups.UniOnFailure.lambda$call$4(UniOnFailure.java:102)
at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:92)
... 42 more
Caused by: io.vertx.core.impl.NoStackTraceThrowable: Timeout
and
2022-01-30 16:29:53,581 ERROR [com.lau.lin.MyService] (vert.x-eventloop-thread-10) Persisting failed: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
[Exception 0] java.lang.IllegalStateException: HR000061: Session is currently connecting to database
[Exception 1] java.lang.IllegalStateException: HR000061: Session is currently connecting to database
at io.smallrye.mutiny.groups.UniOnFailure.lambda$call$3(UniOnFailure.java:108)
at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform$UniOnFailureTransformProcessor.onFailure(UniOnFailureTransform.java:54)
at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onFailure(UniOperatorProcessor.java:48)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forwardResult(UniCreateFromCompletionStage.java:60)
at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
at java.base/java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:887)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2325)
at java.base/java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:144)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage$CompletionStageUniSubscription.forward(UniCreateFromCompletionStage.java:51)
at io.smallrye.mutiny.operators.uni.builders.UniCreateFromCompletionStage.subscribe(UniCreateFromCompletionStage.java:35)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnItemTransformToUni.subscribe(UniOnItemTransformToUni.java:25)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnFailureTransform.subscribe(UniOnFailureTransform.java:31)
at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.performInnerSubscription(UniOnFailureFlatMap.java:101)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.dispatch(UniOnFailureFlatMap.java:83)
at io.smallrye.mutiny.operators.uni.UniOnFailureFlatMap$UniOnFailureFlatMapProcessor.onFailure(UniOnFailureFlatMap.java:60)
16:30:23: Execution finished 'quarkusDev'.
What am I missing?
Related
I am trying to access the read replica database from Spring R2DBC.My connection string looks like this
spring:
r2dbc:
url: r2dbc:mysql://db-master-dev-pvt.xyz***.com:3306,db-replica-dev-pvt.xyz**.com:3306/employee?autoReconnect=true&useUnicode=yes&characterEncoding=UTF-8
username:
password:
but I am getting an unknown host. I am following the below document
https://r2dbc.io/spec/0.8.2.RELEASE/spec/html/#overview.connection.discovery
as per Documentation, we can multiple host configurations separated by comma (,) but when I have tried to do a query or do any health check it's throwing an unknown host exception. Same configuration working fine with Spring Data JPA.
{
"database": "MySQL",
"validationQuery": "validate(REMOTE)",
"error": "java.net.UnknownHostException: failed to resolve 'db-master-dev-pvt.xyz**.com:3306,db-replica-dev-pvt.xyz**.com:3306'"
}
Stack Trace
{"#timestamp":"2021-02-12T11:34:18.438Z","#version":"1","message":"Operator called default onErrorDropped","logger_name":"reactor.core.publisher.Operators","thread_name":"reactor-tcp-epoll-1","level":"ERROR","level_value":40000,"stack_trace":"java.net.UnknownHostException: failed to resolve 'myDB-master-dev-pvt.xyz**.com:3306,myDB-replica-dev-pvt.myAPI.com:3306' after 2 queries \n\tat io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1013)\n\t... 35 common frames omitted\nWrapped by: org.springframework.transaction.CannotCreateTransactionException: Could not open R2DBC Connection for transaction; nested exception is java.net.UnknownHostException: failed to resolve 'myDB-master-dev-pvt.myAPI.com:3306,MyDB-replica-dev-pvt.myAPI.com:3306' after 2 queries \n\tat org.springframework.r2dbc.connection.R2dbcTransactionManager.lambda$null$5(R2dbcTransactionManager.java:226)\n\tat reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onError(FluxOnErrorResume.java:94)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxRetry$RetrySubscriber.onError(FluxRetry.java:94)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxPeekFuseable$PeekFuseableSubscriber.onError(FluxPeekFuseable.java:234)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxPeek$PeekSubscriber.onError(FluxPeek.java:221)\n\tat reactor.pool.AbstractPool$Borrower.fail(AbstractPool.java:427)\n\tat reactor.pool.SimpleDequePool.lambda$drainLoop$5(SimpleDequePool.java:309)\n\tat reactor.core.publisher.FluxDoOnEach$DoOnEachSubscriber.onError(FluxDoOnEach.java:186)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.onError(MonoFlatMap.java:172)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.FluxMap$MapSubscriber.onError(FluxMap.java:132)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoCreate$DefaultMonoSink.error(MonoCreate.java:189)\n\tat reactor.netty.resources.NewConnectionProvider$DisposableConnect.onError(NewConnectionProvider.java:139)\n\tat org.springframework.cloud.sleuth.instrument.reactor.ScopePassingSpanSubscriber.onError(ScopePassingSpanSubscriber.java:95)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapMain.secondError(MonoFlatMap.java:192)\n\tat reactor.core.publisher.MonoFlatMap$FlatMapInner.onError(MonoFlatMap.java:259)\n\tat reactor.netty.transport.TransportConnector$MonoChannelPromise.tryFailure(TransportConnector.java:464)\n\tat reactor.netty.transport.TransportConnector.lambda$doResolveAndConnect$6(TransportConnector.java:271)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:109)\n\tat io.netty.resolver.InetSocketAddressResolver$1.operationComplete(InetSocketAddressResolver.java:62)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\tat io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)\n\tat io.netty.resolver.dns.DnsNameResolver.tryFailure(DnsNameResolver.java:936)\n\tat io.netty.resolver.dns.DnsNameResolver.access$500(DnsNameResolver.java:90)\n\tat io.netty.resolver.dns.DnsNameResolver$5.operationComplete(DnsNameResolver.java:956)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\tat io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)\n\tat io.netty.resolver.dns.DnsResolveContext.finishResolve(DnsResolveContext.java:1021)\n\tat io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:966)\n\tat io.netty.resolver.dns.DnsResolveContext.query(DnsResolveContext.java:414)\n\tat io.netty.resolver.dns.DnsResolveContext.tryToFinishResolve(DnsResolveContext.java:938)\n\tat io.netty.resolver.dns.DnsResolveContext.access$700(DnsResolveContext.java:63)\n\tat io.netty.resolver.dns.DnsResolveContext$2.operationComplete(DnsResolveContext.java:467)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)\n\tat io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)\n\tat io.netty.resolver.dns.DnsQueryContext.trySuccess(DnsQueryContext.java:201)\n\tat io.netty.resolver.dns.DnsQueryContext.finish(DnsQueryContext.java:193)\n\tat io.netty.resolver.dns.DnsNameResolver$DnsResponseHandler.channelRead(DnsNameResolver.java:1230)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.epoll.EpollDatagramChannel.read(EpollDatagramChannel.java:681)\n\tat io.netty.channel.epoll.EpollDatagramChannel.access$100(EpollDatagramChannel.java:58)\n\tat io.netty.channel.epoll.EpollDatagramChannel$EpollDatagramChannelUnsafe.epollInReady(EpollDatagramChannel.java:499)\n\tat io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.lang.Thread.run(Thread.java:748)\nWrapped by: reactor.core.Exceptions$ErrorCallbackNotImplemented: org.springframework.transaction.CannotCreateTransactionException: Could not open R2DBC Connection for transaction; nested exception is java.net.UnknownHostException: failed to resolve 'myDB-master-dev-pvt.xyz**.com:3306,myDB-replica-dev-pvt.xyz**.com:3306' after 2 queries \n","caller_class_name":"reactor.util.Loggers$Slf4JLogger","caller_method_name":"error","caller_file_name":"Loggers.java","caller_line_number":314,"traceId":"","instance_activeProfiles":"dev","instance_port":"8080","instance_ip":"instance_ip_IS_UNDEFINED","instance_application_name":"employee-adjustment-service"}
Thanks
I have done the analysis, I am providing that here so that in future if any one want to check on mysql they must see this before planing to R2DBC. I went to the R2DBC team and after that, I found that driver which I am using that don't have multi-host . and Officaily they support on postgres and MSSQL and H2,
I am using below driver
<!-- https://mvnrepository.com/artifact/dev.miku/r2dbc-mysql -->
<dependency>
<groupId>dev.miku</groupId>
<artifactId>r2dbc-mysql</artifactId>
<version>0.8.2.RELEASE</version>
</dependency>
so this need to be fixed in this driver I have to open an issue there
https://github.com/mirromutth/r2dbc-mysql/issues/169 in order to fix it.
Thanks, Mark #mp911de from Spring R2DBC for clarification and support.
I`m able to invoke with multiple hosts comma seperated hosts
r2dbc.oracle.config.host1=a.b.com
r2dbc.oracle.config.host2=x.y.com
And My config class goes as below
#Value("${r2dbc.oracle.config.host1}")
private String host1;
#Value("${r2dbc.oracle.config.host2}")
private String host2;
#Bean()
#Qualifier("remoteabcConnectionFactory")
public ConnectionFactory remoteRemedyConnectionFactory() {
String hosts = host1 + "," + host2;
return ConnectionFactories.get(ConnectionFactoryOptions.builder()
.option(DRIVER, driver)
.option(HOST, host)
.option(HOST, hosts)
.option(PORT, Integer.valueOf(port))
.option(USER, user)
.option(PASSWORD, password)
.option(DATABASE, database)
.build());
}
You can also get comma separated hosts from one propertt param in
app.properties
I want to create a rsocket channel where the data sent from the server can be either a reaction to a client request or a push. I use a flux merge for that.
It's referential data : the refresh can be asked by the client and the server can also push updates.
So I have this on the server side :
#MessageMapping("update-stream")
Flux<DomainObject> addUpdatesListener(Flux<RefreshRequest> requests) {
Flux<DomainObject> pushFlux = Flux.from(this.flux)
.doOnError((e) -> log.error("Error on push flux : {}", e, e));
return requests
.map(this::getUpdates)
.flatMap(Flux::fromIterable)
.doOnError((e) -> log.error("Error on channel flux : {}", e, e))
.mergeWith(pushFlux)
.doOnError((e) -> log.error("Error on merged flux : {}", e, e));
}
It works excepts that when I stop the client I have the following error :
06-07-2020 15:58:53.168 [reactor-http-nio-3] ERROR reactor.core.publisher.Operators.error - Operator called default onErrorDropped
java.util.concurrent.CancellationException: Disposed
at reactor.core.publisher.FluxProcessor.dispose(FluxProcessor.java:80)
at io.rsocket.core.RSocketResponder$3.hookOnCancel(RSocketResponder.java:513)
at reactor.core.publisher.BaseSubscriber.cancel(BaseSubscriber.java:230)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at io.rsocket.core.RSocketResponder.cleanUpSendingSubscriptions(RSocketResponder.java:275)
at io.rsocket.core.RSocketResponder.cleanup(RSocketResponder.java:265)
at io.rsocket.core.RSocketResponder.tryTerminate(RSocketResponder.java:167)
at io.rsocket.core.RSocketResponder.tryTerminateOnConnectionClose(RSocketResponder.java:160)
at reactor.core.publisher.LambdaMonoSubscriber.onComplete(LambdaMonoSubscriber.java:132)
at reactor.core.publisher.MonoProcessor$NextInner.onComplete(MonoProcessor.java:518)
at reactor.core.publisher.MonoProcessor.onNext(MonoProcessor.java:308)
at reactor.core.publisher.MonoProcessor.onComplete(MonoProcessor.java:265)
at io.rsocket.internal.BaseDuplexConnection.dispose(BaseDuplexConnection.java:23)
at io.rsocket.transport.netty.TcpDuplexConnection.lambda$new$0(TcpDuplexConnection.java:60)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:577)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:570)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:549)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:490)
at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:615)
at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:604)
at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at io.netty.channel.AbstractChannel$CloseFuture.setClosed(AbstractChannel.java:1158)
at io.netty.channel.AbstractChannel$AbstractUnsafe.doClose0(AbstractChannel.java:760)
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:736)
at io.netty.channel.AbstractChannel$AbstractUnsafe.close(AbstractChannel.java:607)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.closeOnRead(AbstractNioByteChannel.java:105)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:171)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
If I don't do the merge, I have no error.
I tried many different versions but I cant find a way to have both the push an no error logged on client quit.
What am I missing ?
Thank a lot.
The problem disapears when upgrading from spring-boot 2.3.0.RELEASE to 2.3.1.RELEASE.
We are using REDIS cache , and using Spring-Redis module , we set the maxActiveConnections 10 in application configuration , but sometimes in my applications am seeing below errors
Exception occurred while querying cache : org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
is it because of in the Redis server their are no more connections to give to my applications or any other reason , can anyone please suggest on this ?
Note : their are 15 applications which are using the same Redis server to store the data , i mean 15 applications need connections from this single redis server only , for now we set 10 as maxActiveConnections for each of the 15 applications
To check how many clients are connected to redis you can use redis-cli and type this command: redis> INFO more specifically info Clients command.
192.168.8.176:8023> info Clients
# Clients
connected_clients:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
Form Jedis source code, it seems that the exception happened for the following reason:
Exhausted cache: // The exception was caused by an exhausted pool
or // Otherwise, the exception was caused by the implemented activateObject() or ValidateObject()
Here is the code snippet of Jedis getResource method:
public T getResource() {
try {
return internalPool.borrowObject();
} catch (NoSuchElementException nse) {
if (null == nse.getCause()) { // The exception was caused by an exhausted pool
throw new JedisExhaustedPoolException(
"Could not get a resource since the pool is exhausted", nse);
}
// Otherwise, the exception was caused by the implemented activateObject() or ValidateObject()
throw new JedisException("Could not get a resource from the pool", nse);
} catch (Exception e) {
throw new JedisConnectionException("Could not get a resource from the pool", e);
}
}
I am trying to use HBase for building some real time API's. Hence my use case is to support ~10000 concurrent requests per second. I am trying to do some connection pooling so as to achieve multi thread access. I followed this documentation to create the connection: https://hbase.apache.org/1.1/apidocs/org/apache/hadoop/hbase/client/ConnectionFactory.html
But I keep getting this error when I make concurrent requests to my API:
WARN [http-nio-34000-exec-93-SendThread(d-3zjyk02.target.com:2181)]
19 Apr 2017 04:48:13:872 (ClientCnxn.java:1102) - Session 0x0 for
server d-3zjyk02.target.com/10.66.241.30:2181, unexpected error,
closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
Here is how I am creating the connection:
// Connection to the cluster. A single connection shared by all application threads
private Connection connection = null;
public Connection getHBaseConnection() throws Exception {
if (connection == null) {
try {
Configuration configuration = HBaseConfiguration.create();
configuration.addResource("core-site.xml");
configuration.addResource("hbase-site.xml");
configuration.addResource("hdfs-site.xml");
connection = ConnectionFactory.createConnection(configuration);
} catch (Exception ex) {
LOG.error("Exception in creating the HBase connection object: " + ex.getMessage());
throw new Exception("Exception in creating the HBase connection: " + ex.getMessage());
}
}
return connection;
}
And here is how I use the get HBase connection method to some scan operations:
try {
connection = getHBaseConnection();
afterConnectionStartTime = System.currentTimeMillis();
LOG.info("[" + (System.currentTimeMillis() - startTime) + "]ms" + " ...TIME TAKEN to get the HBase connection object");
if (connection != null) {
table = connection.getTable(TableName.valueOf(TABLE_NAME));
Scan scan = new Scan(Bytes.toBytes(rowKeyStartDate), Bytes.toBytes(rowKeyEndDate));
scan.addColumn(COLUMN_FAMILY, ITEM);
}
This code works fine for any number of sequential requests, but when I do concurrent requests, I keep getting this error.
Some of the observations from my research on this issue:
1) This error is related to zookeeper closing the socket after certain number of requests (which I assume when it exceeds the max client connections (40) mentioned in my zoo.cfg file). But what I don't understand is why the concurrent requests are going to zookeeper in the first place. The first request should open the connection object and all the subsequent requests should use that pre existing connection to directly talk to region servers.
2) I am assuming this is the right way to do the connection pooling (at least as per the official Hbase doc). If no, whats the right way to do it?
3) I don't want to increase the max client connections in the zookeeper cfg file, thought it might be a temporary hack that can do my work.
Any help / suggestions is much appreciated.
Thanks!
I'm trying to use JMS Point-to-Point in JMeter to send messages to a SI bus queue in Websphere.
I'm fairly certain I have the following correct:
QueueConnection Factory
JNDI name Request queue
JNDI Properties, Initial Context Factory set to com.ibm.websphere.naming.WsnInitialContextFactory
I have guessed the URL:
iiop://ipaddressofWebSphereServer:2809
When I run the sampler I get the following error in the jmeter.log:
2013/03/14 09:09:01 ERROR - jmeter.protocol.jms.sampler.JMSSampler: Error getting WsnNameService properties javax.naming.NamingException: Error getting WsnNameService properties [Root exception is org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible vmcid: IBM minor code: E07 completed: No]
at com.ibm.ws.naming.util.WsnInitCtxFactory.mergeWsnNSProperties(WsnInitCtxFactory.java:1444)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootContextFromServer(WsnInitCtxFactory.java:951)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootJndiContext(WsnInitCtxFactory.java:866)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getInitialContextInternal(WsnInitCtxFactory.java:546)
at com.ibm.ws.naming.util.WsnInitCtx.getContext(WsnInitCtx.java:123)
at com.ibm.ws.naming.util.WsnInitCtx.getContextIfNull(WsnInitCtx.java:798)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:164)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
at javax.naming.InitialContext.lookup(InitialContext.java:436)
at org.apache.jmeter.protocol.jms.sampler.JMSSampler.threadStarted(JMSSampler.java:303)
at org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:597)
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1001)
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1002)
at org.apache.jorphan.collections.HashTree.traverse(HashTree.java:986)
at org.apache.jmeter.threads.JMeterThread.threadStarted(JMeterThread.java:566)
at org.apache.jmeter.threads.JMeterThread.initRun(JMeterThread.java:554)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:252)
at java.lang.Thread.run(Thread.java:736)
Caused by: org.omg.CORBA.TRANSIENT: initial and forwarded IOR inaccessible vmcid: IBM minor code: E07 completed: No
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1270)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1330)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1158)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1296)
at com.ibm.rmi.corba.ClientDelegate.request(ClientDelegate.java:1877)
at com.ibm.CORBA.iiop.ClientDelegate.request(ClientDelegate.java:1252)
at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:458)
at com.ibm.WsnBootstrap._WsnNameServiceStub.getProperties(_WsnNameServiceStub.java:38)
at com.ibm.ws.naming.util.WsnInitCtxFactory.mergeWsnNSProperties(WsnInitCtxFactory.java:1441)
... 17 more
Caused by: java.net.UnknownHostException: ipaddressofWebSphereServer
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:207)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)
at java.net.Socket.connect(Socket.java:539)
at java.net.Socket.connect(Socket.java:488)
at java.net.Socket.<init>(Socket.java:385)
at java.net.Socket.<init>(Socket.java:199)
at com.ibm.ws.orbimpl.transport.WSTCPTransportConnection.createSocket(WSTCPTransportConnection.java:270)
at com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.java:354)
at com.ibm.ws.orbimpl.transport.WSTransport.getConnection(WSTransport.java:436)
at com.ibm.CORBA.transport.TransportBase.getConnection(TransportBase.java:187)
at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:89)
at com.ibm.rmi.iiop.GIOPImpl.getConnection(GIOPImpl.java:130)
at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:219)
at com.ibm.rmi.corba.ClientDelegate.locate(ClientDelegate.java:1974)
at com.ibm.rmi.corba.ClientDelegate._createRequest(ClientDelegate.java:1999)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1180)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1266)
... 25 more
2013/03/14 09:09:01 WARN - jmeter.protocol.jms.sampler.JMSSampler: Session may not be null while creating message java.lang.IllegalStateException: Session may not be null while creating message
at org.apache.jmeter.protocol.jms.sampler.JMSSampler.createMessage(JMSSampler.java:179)
at org.apache.jmeter.protocol.jms.sampler.JMSSampler.sample(JMSSampler.java:140)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:428)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:256)
at java.lang.Thread.run(Thread.java:736)
And I get this error in the orbtrc log:
09:09:01.878 com.ibm.ws.orbimpl.transport.WSTCPTransportConnection connect:403 Thread Group 1-1 ORBRas[default] java.net.UnknownHostException: <i><b> ipaddressofWebSphereServer</i></b>
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:207)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)
at java.net.Socket.connect(Socket.java:539)
at java.net.Socket.connect(Socket.java:488)
at java.net.Socket.<init>(Socket.java:385)
at java.net.Socket.<init>(Socket.java:199)
at com.ibm.ws.orbimpl.transport.WSTCPTransportConnection.createSocket(WSTCPTransportConnection.java:270)
at com.ibm.CORBA.transport.TransportConnectionBase.connect(TransportConnectionBase.java:354)
at com.ibm.ws.orbimpl.transport.WSTransport.getConnection(WSTransport.java:436)
at com.ibm.CORBA.transport.TransportBase.getConnection(TransportBase.java:187)
at com.ibm.rmi.iiop.TransportManager.get(TransportManager.java:89)
at com.ibm.rmi.iiop.GIOPImpl.getConnection(GIOPImpl.java:130)
at com.ibm.rmi.iiop.GIOPImpl.locate(GIOPImpl.java:219)
at com.ibm.rmi.corba.ClientDelegate.locate(ClientDelegate.java:1974)
at com.ibm.rmi.corba.ClientDelegate._createRequest(ClientDelegate.java:1999)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1180)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1266)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1330)
at com.ibm.rmi.corba.ClientDelegate.createRequest(ClientDelegate.java:1158)
at com.ibm.CORBA.iiop.ClientDelegate.createRequest(ClientDelegate.java:1296)
at com.ibm.rmi.corba.ClientDelegate.request(ClientDelegate.java:1877)
at com.ibm.CORBA.iiop.ClientDelegate.request(ClientDelegate.java:1252)
at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:458)
at com.ibm.WsnBootstrap._WsnNameServiceStub.getProperties(_WsnNameServiceStub.java:38)
at com.ibm.ws.naming.util.WsnInitCtxFactory.mergeWsnNSProperties(WsnInitCtxFactory.java:1441)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootContextFromServer(WsnInitCtxFactory.java:951)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getRootJndiContext(WsnInitCtxFactory.java:866)
at com.ibm.ws.naming.util.WsnInitCtxFactory.getInitialContextInternal(WsnInitCtxFactory.java:546)
at com.ibm.ws.naming.util.WsnInitCtx.getContext(WsnInitCtx.java:123)
at com.ibm.ws.naming.util.WsnInitCtx.getContextIfNull(WsnInitCtx.java:798)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:164)
at com.ibm.ws.naming.util.WsnInitCtx.lookup(WsnInitCtx.java:179)
at javax.naming.InitialContext.lookup(InitialContext.java:436)
at org.apache.jmeter.protocol.jms.sampler.JMSSampler.threadStarted(JMSSampler.java:303)
at org.apache.jmeter.threads.JMeterThread$ThreadListenerTraverser.addNode(JMeterThread.java:597)
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1001)
at org.apache.jorphan.collections.HashTree.traverseInto(HashTree.java:1002)
at org.apache.jorphan.collections.HashTree.traverse(HashTree.java:986)
at org.apache.jmeter.threads.JMeterThread.threadStarted(JMeterThread.java:566)
at org.apache.jmeter.threads.JMeterThread.initRun(JMeterThread.java:554)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:252)
at java.lang.Thread.run(Thread.java:736)