Throwing exception in MultiTenantConnectionProvider, exhausts the connections in Connection Pool - spring

I am using multi tenancy by schema in MySQL as,
class SchemaPerTenantConnectionProvider : MultiTenantConnectionProvider {
#Autowired
private lateinit var dataSource: DataSource
#Throws(SQLException::class)
override fun getAnyConnection() = this.dataSource.connection
#Throws(SQLException::class)
override fun releaseAnyConnection(connection: Connection) {
connection.close()
}
#Throws(SQLException::class)
override fun getConnection(tenantIdentifier: String): Connection {
val connection = this.anyConnection
try {
connection.createStatement().execute("USE $tenantIdentifier ")
} catch (e: SQLException) {
throw SQLException("Could not alter JDBC connection to schema [$tenantIdentifier]")
}
return connection
}
...
}
My connection pool size is 10, now if any invalid tenantIdentifier is passed 10 times, 10 connections are exhausted, after that application is unable to acquire any connection.
Tried throwing Exception, HibernateException and it didn't help. Using connection with default schema will fetch wrong results. Is there a way to handle this scenario in getConnection(), to not to exhaust connection limits?

Below configuration should work, overriding public void releaseConnection(String tenantIdentifier, Connection connection) will ensure connection get released back to the connection pool.
public class MultiTenantConnectionProviderImpl
implements MultiTenantConnectionProvider, Stoppable {
private final ConnectionProvider connectionProvider = ConnectionProviderUtils.buildConnectionProvider( "master" );
#Override
public Connection getAnyConnection() throws SQLException {
return connectionProvider.getConnection();
}
#Override
public void releaseAnyConnection(Connection connection) throws SQLException {
connectionProvider.closeConnection( connection );
}
#Override
public Connection getConnection(String tenantIdentifier) throws SQLException {
final Connection connection = getAnyConnection();
try {
connection.createStatement().execute( "USE " + tenanantIdentifier );
}
catch ( SQLException e ) {
throw new HibernateException(
"Could not alter JDBC connection to specified schema [" +
tenantIdentifier + "]",
e
);
}
return connection;
}
#Override
public void releaseConnection(String tenantIdentifier, Connection connection) throws SQLException {
try {
connection.createStatement().execute( "USE master" );
}
catch ( SQLException e ) {
// on error, throw an exception to make sure the connection is not returned to the pool.
// your requirements may differ
throw new HibernateException(
"Could not alter JDBC connection to specified schema [" +
tenantIdentifier + "]",
e
);
}
connectionProvider.closeConnection( connection );
}
...
}
Next, fine tuning the datasource configuration in spring boot:
# Number of ms to wait before throwing an exception if no connection is available.
spring.datasource.tomcat.max-wait=10000
# Maximum number of active connections that can be allocated from this pool at the same time.
spring.datasource.tomcat.max-active=50
Reference : Working with datasources
If the issue still persist, go ahead with datasource connection pooling mechanism support such as Hikari etc.

closing connection, in case of error solved the problem.
#Throws(SQLException::class)
override fun getConnection(tenantIdentifier: String): Connection {
val connection = this.anyConnection
try {
connection.createStatement().execute("USE $tenantIdentifier ")
} catch (e: SQLException) {
connection.close()
throw SQLException("Could not alter JDBC connection to schema [$tenantIdentifier]")
}
return connection
}

Related

Intermittent SocketTimeoutException with elasticsearch-rest-client-7.2.0

I am using RestHighLevelClient version 7.2 to connect to the ElasticSearch cluster version 7.2. My cluster has 3 Master nodes and 2 data nodes. Data node memory config: 2 core and 8 GB. I have used to below code in my spring boot project to create RestHighLevelClient instance.
#Bean(destroyMethod = "close")
#Qualifier("readClient")
public RestHighLevelClient readClient(){
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(elasticUser, elasticPass));
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, elasticPort))
.setHttpClientConfigCallback(httpClientBuilder ->httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider).setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()));
builder.setRequestConfigCallback(requestConfigBuilder -> requestConfigBuilder.setConnectTimeout(30000).setSocketTimeout(60000)
);
RestHighLevelClient restClient = new RestHighLevelClient(builder);
return restClient;
}
RestHighLevelClient is a singleton bean. Intermittently I am getting SocketTimeoutException with both GET and PUT request. The index size is around 50 MB. I have tried increasing the socket timeout value, but still, I receive the same error. Am I missing some configuration? Any help would be appreciated.
I got the issue just wanted to share so that it can help others.
I was using Load Balancer to connect to the ElasticSerach Cluster.
As you can see from my RestClientBuilder code that I was using only the loadbalancer host and port. Although I have multiple master node, still RestClient was not retrying my request in case of connection timeout.
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, elasticPort))
.setHttpClientConfigCallback(httpClientBuilder ->httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider).setDefaultIOReactorConfig(IOReactorConfig.custom().setIoThreadCount(5).build()));
According to the RestClient code if we use a single host then it won't retry in case of any connection issue.
So I changed my code as below and it started working.
RestClientBuilder builder = RestClient.builder(new HttpHost(elasticHost, 9200),new HttpHost(elasticHost, 9201))).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
For complete RestClient code please refer https://github.com/elastic/elasticsearch/blob/master/client/rest/src/main/java/org/elasticsearch/client/RestClient.java
Retry code block in RestClient
private Response performRequest(final NodeTuple<Iterator<Node>> nodeTuple,
final InternalRequest request,
Exception previousException) throws IOException {
RequestContext context = request.createContextForNextAttempt(nodeTuple.nodes.next(), nodeTuple.authCache);
HttpResponse httpResponse;
try {
httpResponse = client.execute(context.requestProducer, context.asyncResponseConsumer, context.context, null).get();
} catch(Exception e) {
RequestLogger.logFailedRequest(logger, request.httpRequest, context.node, e);
onFailure(context.node);
Exception cause = extractAndWrapCause(e);
addSuppressedException(previousException, cause);
if (nodeTuple.nodes.hasNext()) {
return performRequest(nodeTuple, request, cause);
}
if (cause instanceof IOException) {
throw (IOException) cause;
}
if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
}
throw new IllegalStateException("unexpected exception type: must be either RuntimeException or IOException", cause);
}
ResponseOrResponseException responseOrResponseException = convertResponse(request, context.node, httpResponse);
if (responseOrResponseException.responseException == null) {
return responseOrResponseException.response;
}
addSuppressedException(previousException, responseOrResponseException.responseException);
if (nodeTuple.nodes.hasNext()) {
return performRequest(nodeTuple, request, responseOrResponseException.responseException);
}
throw responseOrResponseException.responseException;
}
I'm facing the same issue, and seeing this I realized that the retry is happening on my side too in each host (I have 3 host and the exception happens in 3 threads). I wanted to post it since you might face the same issue or someone else might come to this post because of the same SocketConnection Exception.
Searching the official docs, the HighLevelRestClient uses under the hood the RestClient, and the RestClient uses CloseableHttpAsyncClient which have a connection pool. ElasticSearch specifies that you should close the connection once that you are done, (which sounds ambiguous the definition of "done" in an application), but in general in internet I have found that you should close it when the application is closing or ending, rather than when you finished querying.
Now on the official documentation of apache they have an example to handle the connection pool, which i'm trying to follow, I'll try to replicate the scenario and will post if that fixes my issue, the code can be found here:
https://hc.apache.org/httpcomponents-asyncclient-dev/httpasyncclient/examples/org/apache/http/examples/nio/client/AsyncClientEvictExpiredConnections.java
This is what i have so far:
#Bean(name = "RestHighLevelClientWithCredentials", destroyMethod = "close")
public RestHighLevelClient elasticsearchClient(ElasticSearchClientConfiguration elasticSearchClientConfiguration,
RestClientBuilder.HttpClientConfigCallback httpClientConfigCallback) {
return new RestHighLevelClient(
RestClient
.builder(getElasticSearchHosts(elasticSearchClientConfiguration))
.setHttpClientConfigCallback(httpClientConfigCallback)
);
}
#Bean
#RefreshScope
public RestClientBuilder.HttpClientConfigCallback getHttpClientConfigCallback(
PoolingNHttpClientConnectionManager poolingNHttpClientConnectionManager,
CredentialsProvider credentialsProvider
) {
return httpAsyncClientBuilder -> {
httpAsyncClientBuilder.setSSLHostnameVerifier(NoopHostnameVerifier.INSTANCE);
httpAsyncClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
httpAsyncClientBuilder.setConnectionManager(poolingNHttpClientConnectionManager);
return httpAsyncClientBuilder;
};
}
public class ElasticSearchClientManager {
private ElasticSearchClientManager.IdleConnectionEvictor idleConnectionEvictor;
/**
* Custom client connection manager to create a connection watcher
*
* #param elasticSearchClientConfiguration elasticSearchClientConfiguration
* #return PoolingNHttpClientConnectionManager
*/
#Bean
#RefreshScope
public PoolingNHttpClientConnectionManager getPoolingNHttpClientConnectionManager(
ElasticSearchClientConfiguration elasticSearchClientConfiguration
) {
try {
SSLIOSessionStrategy sslSessionStrategy = new SSLIOSessionStrategy(getTrustAllSSLContext());
Registry<SchemeIOSessionStrategy> sessionStrategyRegistry = RegistryBuilder.<SchemeIOSessionStrategy>create()
.register("http", NoopIOSessionStrategy.INSTANCE)
.register("https", sslSessionStrategy)
.build();
ConnectingIOReactor ioReactor = new DefaultConnectingIOReactor();
PoolingNHttpClientConnectionManager poolingNHttpClientConnectionManager =
new PoolingNHttpClientConnectionManager(ioReactor, sessionStrategyRegistry);
idleConnectionEvictor = new ElasticSearchClientManager.IdleConnectionEvictor(poolingNHttpClientConnectionManager,
elasticSearchClientConfiguration);
idleConnectionEvictor.start();
return poolingNHttpClientConnectionManager;
} catch (IOReactorException e) {
throw new RuntimeException("Failed to create a watcher for the connection pool");
}
}
private SSLContext getTrustAllSSLContext() {
try {
return new SSLContextBuilder()
.loadTrustMaterial(null, (x509Certificates, string) -> true)
.build();
} catch (Exception e) {
throw new RuntimeException("Failed to create SSL Context with open certificate", e);
}
}
public IdleConnectionEvictor.State state() {
return idleConnectionEvictor.evictorState;
}
#PreDestroy
private void finishManager() {
idleConnectionEvictor.shutdown();
}
public static class IdleConnectionEvictor extends Thread {
private final NHttpClientConnectionManager nhttpClientConnectionManager;
private final ElasticSearchClientConfiguration elasticSearchClientConfiguration;
#Getter
private State evictorState;
private volatile boolean shutdown;
public IdleConnectionEvictor(NHttpClientConnectionManager nhttpClientConnectionManager,
ElasticSearchClientConfiguration elasticSearchClientConfiguration) {
super();
this.nhttpClientConnectionManager = nhttpClientConnectionManager;
this.elasticSearchClientConfiguration = elasticSearchClientConfiguration;
}
#Override
public void run() {
try {
while (!shutdown) {
synchronized (this) {
wait(elasticSearchClientConfiguration.getExpiredConnectionsCheckTime());
// Close expired connections
nhttpClientConnectionManager.closeExpiredConnections();
// Optionally, close connections
// that have been idle longer than 5 sec
nhttpClientConnectionManager.closeIdleConnections(elasticSearchClientConfiguration.getMaxTimeIdleConnections(),
TimeUnit.SECONDS);
this.evictorState = State.RUNNING;
}
}
} catch (InterruptedException ex) {
this.evictorState = State.NOT_RUNNING;
}
}
private void shutdown() {
shutdown = true;
synchronized (this) {
notifyAll();
}
}
public enum State {
RUNNING,
NOT_RUNNING
}
}
}

Facing Too many connection issue on HIkariCP

I have a Java JDBC application, after 100 queries of Select Jetty server crashed and return below error:
ERROR com.zaxxer.hikari.pool.HikariPool - dev-server - Exception during pool initialization.
java.sql.SQLNonTransientConnectionException: Too many connections
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:526)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:513)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:115)
at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:1606)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:633)
at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:347)
at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:219)
at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:358)
at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:201)
at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:443)
at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:514)
at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:111)
at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getDataSource(HikariDataSourceSupport.java:70)
at com.jrg.platform.commons.hikari.HikariDataSourceSupport.getConnection(HikariDataSourceSupport.java:82)
at com.jrg.platform.services.internal.pcms.data.campaign.CampaignAlertDaoImpl.getCampaignAlerts(CampaignAlertDaoImpl.java:40)
at com.jrg.platform.services.internal.pcms.service.CampaignAlertServiceImpl.campaignAlerts(CampaignAlertServiceImpl.java:43)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource.getCampaignAlerts(CampaignAlertResource.java:52)
at com.jrg.platform.services.internal.pcms.resource.CampaignAlertResource_$$_jvstf5a_6._d5getCampaignAlerts(CampaignAlertResource_$$_jvstf5a_6.java)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Unable to find the solution. I am just getting the connection from HikariDataSourceSupport and performing the query.
the sample code of getting connection is given below:
default HikariDataSource getDataSource(E e, boolean readOnly) throws IOException {
String key = this.getKey(e, readOnly);
Map<String, HikariDataSource> sources = this.getDataSources();
if(!sources.containsKey(key)) {
synchronized(this.getMonitor()) {
if(!sources.containsKey(key)) {
if(logger.isDebugEnabled()) {
logger.debug("creating new DataSource for {}", key);
}
Config config = this.getConfig(e, readOnly);
if(!config.hasPathOrNull("jdbcUrl")) {
throw new EnvironmentNotConfigured(e, readOnly);
}
sources.put(key, new HikariDataSource(new HikariConfig(ConfigUtils.asProperties(config))));
}
}
}
return (HikariDataSource)sources.get(key);
}
default Connection getConnection(E env, boolean readOnly) throws SQLException, IOException {
return this.getDataSource(env, readOnly).getConnection();
}
There was problem in configuration HikariCP. When request generated and try to get connection, the create connection method was creating the new datasource because the last state was destroy due to view scope of the bean.
now i have converted that class into to singalton and created the implementation of that interface to inject in the code. it is saving the state of previous data source connection. Now it is working perfect with minimum idle connections.
the code can be seen here:
APP.java
bind(HikariLucktasticDataSourceSupportImpl.class)
.to(HikariDataSourceSupport.class)
.named("hdsSupport")
.in(Singleton.class);
in DAO Layer:
#Inject
#Named("hdsSupport")
private HikariDataSourceSupport hdsSupport;

Mixing JdbcTemplate and raw JDBC

I am experiencing some strange behaviour which I can't easily explain. The following code runs fine:
try (Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.executeUpdate("DELETE FROM product");
} catch (Exception ex) {
throw new RuntimeException(ex);
}
try (Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.executeUpdate("INSERT INTO product ...");
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
However this code causes a deadlock:
jdbcTemplate.update("DELETE FROM product");
try (Connection connection = dataSource.getConnection();
Statement statement = connection.createStatement()) {
statement.executeUpdate("INSERT INTO product ...");
} catch (SQLException ex) {
throw new RuntimeException(ex);
}
The exception is
java.sql.SQLException: Lock wait timeout exceeded; try restarting transaction
Both jdbcTemplate and dataSource are created by Spring boot and autowired
#Autowired
private DataSource dataSource;
#Autowired
private JdbcTemplate jdbcTemplate;
The statements form part of a method in a service (with the #Transactional annotation)
Can anyone explain why this happens?
If you want to use your own JDBC code that plays nice with the connections managed by Spring's transaction management you should use DataSourceUtils to obtain the connection -- see http://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/jdbc/datasource/DataSourceUtils.html#getConnection-javax.sql.DataSource-
In your example, depending on the transaction configuration, the first statement using the JdbcTemplate might not be committed yet, so it would block the next JDBC statement from a different connection. Using the DataSourceUtils, both statements would be using the same connection.

Java Oracle JDBC Connection Timeout

I have developed some JSON web services using Servlets for my mobile app. I'm using (Oracle + Private Tomcat) hosting. I have one single class DBOperations.java which has a lot of static functions which are called in Servets for database operation. I use getConnection() method in each function to get Connection Object, create statement and execute queries. Issue is after some time connection get lost. I'm using the following code to re-establish the connection.
public static Connection conn;
Statement stmt;
public static Connection getConnection() throws SQLException {
if (conn == null || conn.isClosed() ) {
try {
Class.forName("oracle.jdbc.driver.OracleDriver");
conn = DriverManager.getConnection(DB_URL, "username", "password");
return conn;
} catch (ClassNotFoundException | SQLException ex) {
}
} else {
return conn;
}
return conn;
}
I'm unable to figure out how I can handle the timeout/closed connection issue as the above code isn't re-establishing the connection. I need to restart Tomcat to get it back in working state. Any suggestions or help is highly appreciated.
You must use connection pooling, And let Tomcat server to handle everything. Create a JNDI datasource to achieve the same and you will never face such issue.
Used OraceDataSource for connection pooling and it's working perfectly.
public static OracleDataSource ods;
public static Connection getConnection() throws SQLException {
if (ods == null) {
ods = new OracleDataSource();
ods.setURL(DB_URL);
ods.setUser("username");
ods.setPassword("password");
}
return ods.getConnection();
}

java code to connect weblogic datasource to particular target

I have requirement in which i need to connect to datasource each individual target.
If datsource has two target then i need to connect to both target one by one.
I am using below code for datasource connection.
Need to know how to connect to individual target of datasource
public class ConnectWLSDataSource {
public static void main(String[] args) {
Connection conn;
Statement stmt;
ResultSet rs;
String str1;
try {
Properties prop = new Properties();
prop.put(Context.INITIAL_CONTEXT_FACTORY, “weblogic.jndi.WLInitialContextFactory”);
prop.put(Context.PROVIDER_URL, “t3://HOST_NAME:PORT_NUMBER”);
Context ctx = new InitialContext(prop);
Object obj = ctx.lookup(“DATA_SOURCE_NAME”); // java:comp/env/CPDS
System.out.println(“Data Source Found….”);
DataSource ds = (DataSource) obj;
conn = ds.getConnection();
System.out.println(“Data Source User Name::”+conn.getMetaData().getUserName());
stmt = conn.createStatement();
String query = “select 1 from dual”;
System.out.println(“Query ” + query);
rs = stmt.executeQuery(query);
if (rs != null) {
System.out.println(“Some Data Found in Query”);
} else {
System.out.println(“No Data Found in Query”);
}
ctx.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
All you need to do is change the PROVIDER_URL to the specific host you want. For instance we have 4 servers running in a cluster and can choose to use a load balancer if we want to round-robin the connections or we can pick one specific server in the cluster.
we face issue when using this code, and get below exception. in all if 100 requests are serviced one fails with the below exception.
javax.naming.CommunicationException [Root exception is java.net.ConnectException: t3://prod.abc.com:7114: Destination unreachable; nested exception is:
java.net.SocketTimeoutException: Read timed out; No available router to destination].

Resources