JMeter : Check JDBC connection open or close after each thread execution - jdbc

I am trying to check the JDBC connection status after each thread execution whether it is close or open?.
In my thread group there are three things
JDBC connection configuration
JDBC request (select * from employee)
JSR223 PostProcessor
Script :
def connection = org.apache.jmeter.protocol.jdbc.config.DataSourceElement.getConnection('ConnectionString')
log.info('*************Connection closed: '+ connection.isClosed())
Above script is logging the connection status after each thread execution when loop count is 1. Problem here as soon as I change the loop count to >= 2. it started throwing the error below error
Problem in JSR223 script, JSR223 PostProcessor
javax.script.ScriptException: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object
And when I remove the Post processor and increase the loop count it is working fine.
Logs :
2023-02-19 16:15:33,599 INFO o.a.j.t.JMeterThread: Thread started: DB Thread Group 2-1
2023-02-19 16:15:38,054 DEBUG o.a.j.p.j.AbstractJDBCTestElement: executing jdbc:SELECT * FROM EMPLOYEE
2023-02-19 16:15:38,623 INFO o.a.j.e.J.JSR223 PostProcessor: *************Connection closed: false
2023-02-19 16:15:58,637 ERROR o.a.j.e.JSR223PostProcessor: Problem in JSR223 script, JSR223 PostProcessor
javax.script.ScriptException: java.sql.SQLException: Cannot get a connection, pool error Timeout waiting for idle object, borrowMaxWaitDuration=PT10S
at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:320) ~[groovy-jsr223-3.0.11.jar:3.0.11]
at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:71) ~

In the JDBC configuration are you using a connection pool to request connections?
What your test shows is that the JSR223 script is closing the connection, which is probably a good thing from a coding perspective, but the next iteration of the loop tries to execute a request with a closed Connection and blammo. If you switch from raw connections to a connection pool when the JSR 223 closes the connection it'll be returned to the pool and remain open for the next iteration of the loop. You'll have to switch to using DataSource API typically for this, but it's a minor tweak to the script.

I can think of 2 possible reasons:
Either your database is down/overloaded/not reachable via JDBC
Or your connection pool settings need to be tweaked, i.e. max number of connections and/or wait time need to be increased:
In general I don't think your approach is correct, as per JavaDoc:
This method generally cannot be called to determine whether a connection to a database is valid or invalid. A typical client can determine that a connection is invalid by catching any exceptions that might be thrown when an operation is attempted.
So you might want to increase debug logging verbosity for JMeter, your JDBC driver and Java SQL namespace instead

Related

Jmeter: Failing to execute next component after while controller(using CSV data config also)

I have a jmeter test plan with while controller, which loops over URLs from a CSV(CSV contains only URLs):
However the next component 'HTTP Request' never gets executed. in log viewer, I can see this:
2021-02-24 20:23:29,317 INFO o.a.j.t.JMeterThread: Stop Thread seen
for thread Thread Group 1-1, reason:
org.apache.jorphan.util.JMeterStopThreadException: End of
file:C:/Users/Administrator/Downloads/jmeter/urls.csv detected for
CSVDataSet:CSV Data Set Config configured with stopThread:true,
recycle:false
If I set stopThread to False, then this controller never stops. Is there any solution to this?
While controller is configured as follows:
You need to set both "Recycle on EOF" and "Stop thread on EOF" to false because:
if you will be recycling on EOF - the while controller will never stop
if you stop thread on EOF - no elements after the While Controller will be executed
So
Configure your CSV Data Set Config like:
Put your "OWA Inbox Access" sampler under the If Controller and use the following __jexl3() function as the condition:
${__jexl3("${myVar}" != "<EOF>",)}
More information: Using the While Controller in JMeter, see Reading all Values from the CSV and Continue chapter

HikariPool-1 – Connection is not available, request timed out after 30000ms

I am using the standard HikariCP implementation in my SpringBoot 2.0.1 application. But after a while I get the same error over and over
HikariPool-1 – Connection is not available, request timed out after 30000ms
I first checked the code if there are any not closed connection or missing Transactional annotations but I did not find anything. I also tried then to increase the pool and decrease the time out in my application.yml but it seems that this does not have any effect.
The weird thing is that HikariCP seems to create only 4 pooled connection and I am not able to override these properties.
HikariPool-1 - Timeout failure stats (total=4, active=4, idle=0, waiting=100)
This is my application.yml file
spring:
jpa:
hibernate:
ddl-auto: update
datasource:
hikari:
maximum-pool-size: 64
connection-timeout: 5000
Your application is not able to get the connection within the 30s which is the default connection timeout for HikariCP. There can be reasons for that such as:
The Pool size is small, and the requests take too long to execute (more than 30 s).
Possible explanation is that for e.g your pool size is 1 and your one request/query takes approximately 500ms to execute. suppose you have opened for example approximately 200 concurrent query requests, then there might be a chance that some requests will be timed out. (one request has waited for the 30s but a connection was not available for it).
Disclaimer: Numbers are taken randomly to explain and is not tested.
To solve this you might want to increase the connectionTimeout or increase the pool size. ( Also check for the memory requirements).
In application.properties set the values
spring.datasource.hikari.minimumIdle=10
spring.datasource.hikari.maximumPoolSize=10
spring.datasource.hikari.idleTimeout=120000
spring.datasource.hikari.maxLifetime=500000
Official Hikari CP documentation link
There can be another reason for this is that your connection is not getting closed properly use try-with-resources to close your connection
e.g
try(Connection c = dataSource.getConnection()) {
//ToDo: Add your logic statement here
log.debug("[d] Tenant '{}' added.", tenantId);
}

Jco Adapter pooling performance deadlock?

We're running an enterprise scale SAP application with front-end springboot clients connecting via Jco adapter 3.0 on Oracle VM using the connection pool (size 100). We're experiencing unsystematic long-running requests > 10s that are not visible in the SAP application server log, i.e. the bottleneck does not appear to be on SAP side.
Looking at the trace files (level 4) for an example request we can see that the time seems lost when the adapter thread tries to get the client from the pool (other threads continue execution, removed the irrelevant threads for clarity):
[20:05:50:259]: [JCoAPI] JCoContext.isStateful(P-foo-CPIC0) in session ID Client-53-1 returns false
[20:05:50:259]: [JCoAPI] JCoContext.begin(P-foo-CPIC0) in session ID Client-53-1
[20:05:50:259]: [JCoAPI] Started context for session Client-53-1
[20:05:50:259]: [JCoAPI] JCoContext.begin() for destination PFOO_200 (P-foo-CPIC0) on context with id Client-53-1; current state counter is 1
[20:05:50:259]: [JCoAPI] destination PFOO_200 destinationID=P-foo-CPIC0 executes Z_foo sessionID=Client-53-1, threadID=0x35
[20:05:50:259]: [JCoAPI] Context.getConnection on destination PFOO_200 (state: destination = STATEFUL, default = STATELESS)
[20:05:50:259]: [JCoAPI] PoolingFactory.getClient() on pool P-foo-CPIC0
--> time lost here
[20:06:20:840]: [JCoAPI] PoolingFactory.getClient() returns handle [3/84977415]
[20:06:20:840]: [JCoAPI] Context.getConnection on destination PFOO_200 nothing found in the context - got client from ConnectionManager [3/84977415]
[20:06:20:840]: [JCoAPI] JCoClient before execute(Z_foo) on handle [3/84977415]
[20:06:20:840]: [JCoRFC] Executing function Z_foo on handle [3/84977415]
[20:06:20:866]: [JCoAPI] JCoClient after execute(Z_foo) on handle [3/84977415] returns after 26 ms
[20:06:20:866]: [JCoAPI] Context.releaseConnection on destination PFOO_200 [3/84977415]
[20:06:20:867]: [JCoAPI] JCoContext.end(P-foo-CPIC0) in session ID Client-53-1
[20:06:20:867]: [JCoAPI] PoolingFactory.releaseClient() handle [3/84977415] into pool P-foo-CPIC0 [pool size: 3, peak limit: 100, waiting threads: 0, currently used: 1]
[20:06:20:879]: [JCoAPI] Finished context for session Client-53-1
[20:06:20:879]: [JCoAPI] JCoContext.end() for destination PFOO_200 (P-foo-CPIC0) on context with id Client-53-1; current state counter is 0
For a typical request the step is handled in milliseconds.
Are there any known limitations or configurations regarding pool handling for the Jco adapter, either on adapter or on SAP side?
Update we've on Jco adapter 3.0.16 and will double-check 3.0.17 now. DNS seems unlikely since we're monitoring dig/nslookup and they're running without delays.
Which JCo patch level do you use?
Did you try to update to the latest JCo patch level 3.0.17 first?
In your time gap the RFC connection will be opened and the RFC logon will be done, if the pool is empty at that time. Did you have a closer look with a higher trace level, or did you have a look into the RFC trace?
This can be anything from not having a free dialog work process at ABAP side, to SAP system database issues (required for the RFC logon authentication checks), slow response times from the SAP message server (if using load balanced logons), SNC handshake issues (if using SNC) or general network issues with the DNS (try using the IP address instead of a hostname).
Another point worth checking: you say your connection pool has size 100. Is it possible, that your program has more than 100 threads? Then it may happen from time to time, that all connections are currently busy in other threads and the current thread has to wait until a function call in another thread completes and a connection is returned to the pool.
(How long a thread waits on an empty pool can be customized via the "pool wait time" parameter.)

Can MAX_UTILIZATION for PROCESSES reached cause "Unable to get managed connection" Exception?

A JBoss 5.2 application server log was filled with thousands of the following exception:
Caused by: javax.resource.ResourceException: Unable to get managed connection for jdbc_TestDB
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:441)
at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:96)
... 9 more
Caused by: javax.resource.ResourceException: No ManagedConnections available within configured blocking timeout ( 30000 [ms] )
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:311)
at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:689)
at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
... 13 more
I've stripped off the first part of the exception, which is basically our internal JDBC wrapper code which tries to get a DB connection from the pool.
Looking at the Oracle DB side I ran the query:
select resource_name, current_utilization, max_utilization, limit_value
from v$resource_limit
where resource_name in ('sessions', 'processes');
This produced the output:
RESOURCE_NAME CURRENT_UTILIZATION MAX_UTILIZATION LIMIT_VALUE
processes 1387 1500 1500
sessions 1434 1586 2272
Given the fact that that PROCESSES limit of 1500 was reached, would this cause the JBoss exceptions we experienced? I've also been investigating the possibility of connection leaks, but haven't found any evidence of that so far.
What is the recommended course of action here? Is simply increasing the limit a valid solution?
Usually when max_utilization gets the processes value listener will refuse new connections to database. you can see the errors relates to it in alert log. to solve this in database side you should increase the processes parameter.
hmm strange. is it possible, that exception wrapping in JBOSS hides the original error? You should get some sql exception whose text starts with ORA-. Maybe your JDBC wrapper does not handle errors properly.
The recommended actions is to:
check configured size of connection pool against processes sessions Oracle startup paramters.
check Oracles view v$session, especially columns STATUS, LAST_CALL_ET, SQL_ID, PREV_SQL_ID.
translate sql_id(prev_sql_id) into sql_text via v$sql.
if you application has a connection leak, sql_id and pred_sql_id might point you onto a place in your source code, where a connection was used last (i.e. where it was leaked).

How JTA/JTS handle transaction time out issue?

Below is my understand that JTA/ JTS handle transaction time out issue. But I cannot find my document or material to back my understand. Is my understand right? Do u know any material is refer to this issue?
Application Server iterates through all the transactions to check timeout. If a transaction timeout occurs, application server marks roll back for the transaction, and log down the detail. But Application Server neither throws exception nor interrupts the transaction this moment. When the transaction thread continue to attempt to access another transactional resource (like JDBC/ JMS), the transactional resource which implements JTA interface will check roll back flag first before go further. Then at this moment, RollbackException is thrown.
==========
Test Case 1:
Set transaction timeout to 10 secs
I. Transaction begin
II. Sleep 20 secs
III. System out "Sleep end"
Result: Timeout occur at 10th secs, and system out log down the timeout detail, but not throw exception. "Sleep end" will be printed.
==========
Test Case 2:
Set transaction timeout to 10 secs
I. Transaction begin
II. Sleep 20 secs
III. Access db 1st time
IV. Access db 2nd time
V. System out "Sleep end"
Result: Timeout occur at 10th secs, and system out logs down the timeout detail, but not throw exception. Exception throws while access db 1st time. "Sleep end" will not be printed.
==========
Test Case 3:
Set transaction timeout to 10 secs
I. Transaction begin
II. Access db and db deadlock
Result: Timeout occur at 10th secs, and system out logs down the timeout detail. No exception throws, the transaction thread is stuck. So transaction timeout control cannot handle db timeout issue. I am so confused about this..
In my understanding, above behavior should be the same while using spring transaction management(JTA) and EJB. Am I right?
Thanks for ur help!
Tested, and proved that my understand should be correct.
Summarize the result as below:
• Transaction timeout control only affects transactional activities (Ex: access DB/ send JMS message).
• Application server do not interrupt current transaction thread immediately while timeout occurs, instead, application server only log down the detail. Timeout exception will throw while transaction commit or attempt to access next transactional activities.
• DB deadlock issue cannot be handled by transaction timeout control. But DB2 have deadlock prevent mechanism to release the deadlock and roll back transaction for some cases.

Resources