Radis Caching Issue (Time Out issue) - spring

I am facing radis issue. I have implemented centeralized radis cluster approch. But whenever the maximum number of services data get cached in the radis it causes an issue, "Read time out" occured.
Please anyone can give me suggestions what can i do?
Please find below properties:
spring :
redis :
database : ${redis.database}
#host : ${redis.host}
#port : ${redis.port}
password : ${redis.password}
pool :
max-active : 8
max-wait : 10000
max-idle : 8
min-idle : 1
timeout : 10000
redis :
cluster :
nodes : 10.xxx.x.xx:1234,10.xxx.x.xx:12365
database : 0
password : 'xxxxxx'
Error Logs:
2022-02-11 13:59:59,996 ERROR [nio-9120-exec-1008] c.h.l.framework.aop.ControllerAspect - [AGW2202111359590622605456]:
redis.clients.jedis.exceptions.JedisDataException: ERR Error running script (call to f_e78e961698f118a64316964cac1ca9e6008c29da): #user_script:34: #user_script: 34: -OOM command not allowed when used memory > 'maxmemory'.
at redis.clients.jedis.Protocol.processError(Protocol.java:127)
at redis.clients.jedis.Protocol.process(Protocol.java:161)
at redis.clients.jedis.Protocol.read(Protocol.java:215)
at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:340)
at redis.clients.jedis.Connection.getOne(Connection.java:322)
at redis.clients.jedis.BinaryJedis.evalsha(BinaryJedis.java:3142)
at redis.clients.jedis.BinaryJedis.evalsha(BinaryJedis.java:3135)
at redis.clients.jedis.BinaryJedisCluster$119.execute(BinaryJedisCluster.java:1270)
at redis.clients.jedis.JedisClusterCommand.runWithRetries(JedisClusterCommand.java:120)
at redis.clients.jedis.JedisClusterCommand.runBinary(JedisClusterCommand.java:81)
at redis.clients.jedis.BinaryJedisCluster.evalsha(BinaryJedisCluster.java:1272)
at com.hisun.lemon.framework.jedis.JedisClusterConnection.evalSha(JedisClusterConnection.java:67)
at sun.reflect.GeneratedMethodAccessor1795.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.data.redis.core.CloseSuppressingInvocationHandler.invoke(CloseSuppressingInvocationHandler.java:57)
at com.sun.proxy.$Proxy537.evalSha(Unknown Source)

Related

Caused by: java.sql.SQLException: Connection not established : Getting this error while processing more than 1k records [closed]

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 11 months ago.
Improve this question
I am working on a spring boot application, which is processing more than 1k json messages in parallel and updating the database simultaneously with the respective fields.
Max 20 threads are running in parallel for this process.
It's running fine with less than 800 json messages.
For this bulk message scenario, I'm getting below error, whenever there is an DB update:
ERROR o.h.e.jdbc.spi.SqlExceptionHelper - Connection not established
2022-04-04 16:35:22.647 [HikariPool-1 connection closer] DEBUG com.zaxxer.hikari.pool.PoolBase - HikariPool-1 - Closing connection com.informix.jdbc.IfxSqliConnect#43a6bd0c: (connection is dead)
ERROR o.s.o.jpa.EntityManagerFactoryUtils - Failed to release JPA EntityManager
org.hibernate.exception.GenericJDBCException: Unable to release JDBC Connection
at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:47)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.releaseConnection(LogicalConnectionManagedImpl.java:223)
at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.close(LogicalConnectionManagedImpl.java:261)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.close(JdbcCoordinatorImpl.java:175)
at org.hibernate.internal.AbstractSharedSessionContract.close(AbstractSharedSessionContract.java:351)
at org.hibernate.internal.SessionImpl.closeWithoutOpenChecks(SessionImpl.java:376)
at org.hibernate.internal.SessionImpl.close(SessionImpl.java:361)
at org.springframework.orm.jpa.EntityManagerFactoryUtils.closeEntityManager(EntityManagerFactoryUtils.java:427)
at org.springframework.orm.jpa.JpaTransactionManager.doCleanupAfterCompletion(JpaTransactionManager.java:650)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.cleanupAfterCompletion(AbstractPlatformTransactionManager.java:1004)
Caused by: java.sql.SQLException: Connection not established
at com.informix.util.IfxErrMsg.getSQLException(IfxErrMsg.java:408)
at com.informix.jdbc.IfxSqliConnect.rollback(IfxSqliConnect.java:2324)
at com.zaxxer.hikari.pool.ProxyConnection.close(ProxyConnection.java:257)
at java.base/jdk.internal.reflect.GeneratedMethodAccessor230.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at net.bull.javamelody.JdbcWrapper$ConnectionInvocationHandler.invoke(JdbcWrapper.java:202)
at net.bull.javamelody.JdbcWrapper$DelegatingInvocationHandler.invoke(JdbcWrapper.java:300)
at com.sun.proxy.$Proxy138.close(Unknown Source)
at org.hibernate.engine.jdbc.connections.internal.DatasourceConnectionProviderImpl.closeConnection(DatasourceConnectionProviderImpl.java:127)
Data source properties configured:
spring:
datasource:
type: org.apache.tomcat.jdbc.pool.DataSource
tomcat:
init-SQL: select col1 from table ( set {1} ) as x( col1 )
default-transaction-isolation: 1
initial-size: 10
max-wait: 60000
max-active: 200
max-idle: 200
min-idle: 100
test-while-idle: true
test-on-connect: true
test-on-borrow: true
test-on-return: true
remove-abandoned : true
remove-abandoned-timeout: 300
time-between-eviction-runs-millis: 5000
min-evictable-idle-time-millis: 60000
validation-interval: 30000
validation-query: select col1 from table ( set {1} ) as x( col1 )
validation-query-timeout: 10
jpa:
open-in-view: false
database-platform: org.hibernate.dialect.InformixDialect
show-sql: false
properties.hibernate.format_sql: true
Can anyone please suggest/advice ?
Since you are using Hikari for connection pooling
I think you also need to manage properties of hikari
Kindly try to configure using these props:
https://docs.spring.io/spring-boot/docs/current/reference/html/application-properties.html#application-properties.data.spring.datasource.hikari

apache nifi Stateless - not able to set parm of controller service (DBCPConnectionPool 1.10.0)

I am following the NiFi 1.10 stateless guildeline to create a simple process group of executing a sql in mysql db. I have put necessary parm of db controller service to parameter context.
it works well in nifi canvas. Then i add it to registry and prepare a json parm file: stateless-simpledb.json
{
"registryUrl": "http://localhost:18080",
"bucketId": "cac8f127-e328-45c1-a4cb-0e03dc837ceb",
"flowId": "cc2753f2-78f3-4449-a2fd-343dfeaafe15",
"flowVersion": "3",
"parameters": {
"lastIngestId" : "20000",
"mysql-jdbc-driver-name" : "com.mysql.jdbc.Driver",
"db-user" : "root",
"db-password" : "password",
"db-con-url" : "jdbc:mysql://localhost:3306/mms",
"jdbc-jar-path" : "/program/jdbc/mysql-connector-java.jar"
}
}
and run the one-off command:
/program/nifi/bin/nifi.sh stateless RunFromRegistry Once --file /app/poc/nifi-stateless/conf/stateless-simpledb.json
It raise error:
=== FlowFileRepository Type ===
org.apache.nifi.controller.repository.RocksDBFlowFileRepository
org.apache.nifi:nifi-framework-nar:1.10.0 || /program/nifi-1.10.0/work/stateless-nars/nifi-framework-nar-1.10.0.nar-unpacked
org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
org.apache.nifi:nifi-framework-nar:1.10.0 || /program/nifi-1.10.0/work/stateless-nars/nifi-framework-nar-1.10.0.nar-unpacked
org.apache.nifi.controller.repository.VolatileFlowFileRepository
org.apache.nifi:nifi-framework-nar:1.10.0 || /program/nifi-1.10.0/work/stateless-nars/nifi-framework-nar-1.10.0.nar-unpacked
=== End FlowFileRepository types ===
23:32:32.626 [main] INFO org.apache.nifi.stateless.bootstrap.ExtensionDiscovery - Successfully discovered extensions in 4411 milliseconds
23:32:32.633 [main] DEBUG org.apache.nifi.stateless.core.ComponentFactory - Setting context class loader to org.apache.nifi.nar.InstanceClassLoader#50fa5938 (parent = org.apache.nifi.nar.NarClassLoader[/program/nifi-1.10.0/work/stateless-nars/nifi-dbcp-service-nar-1.10.0.nar-unpacked]) to create org.apache.nifi.dbcp.DBCPConnectionPool
23:32:32.647 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input #{jdbc-jar-path} found 1 Parameter references: [org.apache.nifi.parameter.StandardParameterReference#2d3eecda]
23:32:32.650 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input /program/jdbc/mysql-connector-java.jar found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input 500 millis found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input 8 found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input 0 found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input 8 found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input -1 found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input -1 found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input 30 mins found 0 Parameter references: []
23:32:32.651 [main] DEBUG org.apache.nifi.parameter.ExpressionLanguageAwareParameterParser - For input -1 found 0 Parameter references: []
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.bootstrap.RunStatelessNiFi.main(RunStatelessNiFi.java:69)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.StatelessNiFi.main(StatelessNiFi.java:103)
... 5 more
Caused by: java.lang.RuntimeException: Failed to enable Controller Service {id=691ecc97-ff46-3a5e-8aad-37dc568bc247, name=MYSQL-MMS-stateless-test, type=class org.apache.nifi.dbcp.DBCPConnectionPool} because validation failed: ['Database Connection URL' is invalid because Database Connection URL is required, 'Database Driver Class Name' is invalid because Database Driver Class Name is required]
at org.apache.nifi.stateless.core.StatelessControllerServiceLookup.enableControllerServices(StatelessControllerServiceLookup.java:133)
at org.apache.nifi.stateless.core.StatelessFlow.<init>(StatelessFlow.java:153)
at org.apache.nifi.stateless.core.StatelessFlow.createAndEnqueueFromJSON(StatelessFlow.java:469)
at org.apache.nifi.stateless.runtimes.Program.runLocal(Program.java:133)
at org.apache.nifi.stateless.runtimes.Program.launch(Program.java:67)
... 10 more
Seems the apache nifi stateless function failed to set controller service even it's in "process group" scope.
Would anyone has any advice?
As mentioned in the comments, this appears to be a known problem with the validation of controller services.
This can be avoided by using Nifi 1.12 and above as it got fixed in the following jira: https://issues.apache.org/jira/plugins/servlet/mobile#issue/NIFI-7380
Though I am not entirely sure of this, it may also be possible that this simply indicates that your controller service is not configured correctly. This would be worth double checking.

Janusgraph returns "Unknown external index backend: search " building mixed index with elasticsearch in Gremlin

When I attempt to create a mixed index in janusgraph 0.3.0 via Gremlin and Gremlin server I get the "Unknown external index backend: search " error. Elasticsearch is running and I believe the config file is correct (I've included all that below.
gremlin> mgmt.get('graph.janusgraph-version')
==>0.3.0
gremlin>
When I run the following...
gremlin> IsCurrentVersion = mgmt.makePropertyKey('is current version').dataType(Boolean.class).cardinality(SINGLE).make();
==>is current version
gremlin> mgmt.buildIndex('iscurrent', Vertex.class).addKey(IsCurrentVersion).buildMixedIndex("search");
I get...
Unknown external index backend: search
# Settings with mutability GLOBAL_OFFLINE are centrally managed in
# JanusGraph's storage backend. After starting the database for the first
# time, this file's copy of this setting is ignored. Use JanusGraph's
# Management System to read or modify this value after bootstrapping.
index.search.backend=elasticsearch
# The hostname or comma-separated list of hostnames of index backend
# servers. This is only applicable to some index backends, such as
# elasticsearch and solr.
#
# Default: 127.0.0.1
# Data Type: class java.lang.String[]
# Mutability: MASKABLE
index.search.hostname=127.0.0.1
The Gremlin Server startup log has...
4060 [main] INFO com.datastax.driver.core.Cluster - New Cassandra host /127.0.0.1:9042 added
4085 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [search]
4813 [main] INFO org.janusgraph.diskstorage.Backend - Configuring index [backend]
4837 [main] INFO org.janusgraph.diskstorage.Backend - Initiated backend operations thread pool of size 8
4931 [main] INFO org.janusgraph.diskstorage.Backend - Configuring total store cache size: 219574576
Elasticsearch is running as shown by...
curl 'http://localhost:9200/?pretty'
{
"name" : "nfZKXFP",
"cluster_name" : "elasticsearch",
"cluster_uuid" : "6QDueUnoQfmKRwuEG-a3nw",
"version" : {
"number" : "6.0.1",
"build_hash" : "601be4a",
"build_date" : "2017-12-04T09:29:09.525Z",
"build_snapshot" : false,
"lucene_version" : "7.0.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

Elasticsearch reboot, number_of_pending_tasks keeps increasing indefinitely

I had to restart the master elasticsearch, the status was red, then after some time the status went yellow (primary shards get assigned).
Now when I'm doing the query curl http://x.x.x.x/_cluster/health?pretty I can see that the "number_of_pending_tasks" keeps increasing (now it is at 200k)
I had a look at the pending tasks and I can see that it is mainly this tasks that get buffered:
, {
"insert_order" : 58176,
"priority" : "NORMAL",
"source" : "indices_store",
"executing" : false,
"time_in_queue_millis" : 619596,
"time_in_queue" : "10.3m"
},
In the meantime I get the error about a rejected execution due to the queue capacity:
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 200) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler#34c87ed9
How can I solve this?

CPU Starvation detected because of MQ exception

**0002bc2d SibMessage W [:] CWSJY0003W: JMSCC3036: An exception has been delivered to the
connections exception listener: '
Message : com.ibm.msg.client.jms.DetailedJMSException: JMSWMQ1107: A problem with this connection has o
ccurred. An error has occurred with the WebSphere MQ JMS connection. Use the linked exception to determine the cause of this e
rror.
Class : class com.ibm.msg.client.jms.DetailedJMSException
Stack : com.ibm.msg.client.wmq.common.internal.Reason.reasonToException(Reason.java:608)
: com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:236)
: com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:851)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1
023)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1381
)
: com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:310)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.j
ava:100)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
: com.ibm.ws.wmqcsi.workqueue.WorkQueueManagerImpl$WorkQueueRunnable.run(WorkQueueManagerImpl.j
ava:648)
: java.lang.Thread.run(Thread.java:738)
Caused by [1] --> Message : com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with compcode '2' ('MQCC_FAILED'
) reason '2009' ('MQRC_CONNECTION_BROKEN').
Class : class com.ibm.mq.MQException
Stack : com.ibm.msg.client.wmq.common.internal.Reason.createException(Reason.java:223)
: com.ibm.msg.client.wmq.internal.WMQConnection.consumer(WMQConnection.java:851)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.callEventHandler(RemoteAsyncConsume.java:1
023)
: com.ibm.mq.jmqi.remote.internal.RemoteAsyncConsume.driveEventsEH(RemoteAsyncConsume.java:1381)
: com.ibm.mq.jmqi.remote.internal.RemoteDispatchThread.run(RemoteDispatchThread.java:310)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.runTask(WorkQueueItem.java:209)
: com.ibm.msg.client.commonservices.workqueue.SimpleWorkQueueItem.runItem(SimpleWorkQueueItem.j
ava:100)
: com.ibm.msg.client.commonservices.workqueue.WorkQueueItem.run(WorkQueueItem.java:224)
: com.ibm.ws.wmqcsi.workqueue.WorkQueueManagerImpl$WorkQueueRunnable.run(WorkQueueManagerImpl.j
ava:648)
: java.lang.Thread.run(Thread.java:738)
'.
Please can anybody explain what this exception means and why has this caused CPU starvation**
Did you review your own logfile?
JMSWMQ1107: A problem with this connection has occurred. An error has
occurred with the WebSphere MQ JMS connection. Use the linked
exception to determine the cause of this error.
And further down it says:
java.lang.Thread.run(Thread.java:738) Caused by [1] --> Message :
com.ibm.mq.MQException: JMSCMQ0001: WebSphere MQ call failed with
compcode '2' ('MQCC_FAILED' ) reason '2009'
('MQRC_CONNECTION_BROKEN').
Did you look up MQ reason code 2009 in the MQ Information Center? Basically, it means the connection to the queue manager was lost and you need to reconnect.
and why has this caused CPU starvation
Any CPU starvation you are seeing has nothing to do with error but rather your code is not properly handling the error and is probably in a loop racing around and around in circles doing nothing.

Resources