Getting NoNodeAvailableException running multiple tests in Spring Boot + Cassandra - spring-boot

Question

Just upgraded to Spring boot 2.7 using jdk19 and decided to use Cassandra bitnami 3 running in docker with my tests "Junit-5", the error I'm getting is No node was available to execute the query, and it happens to the same test cases every time.
No node was available to execute the query; nested exception is \
com.datastax.oss.driver.api.core.NoNodeAvailableException: \
No node was available to execute the query
here is the code I'm using to connect
var loader = DriverConfigLoader.programmaticBuilder()
.withDuration(DefaultDriverOption.REQUEST_TIMEOUT, Duration.ofMinutes(1))
.withString(DefaultDriverOption.LOAD_BALANCING_POLICY_CLASS,
DcInferringLoadBalancingPolicy.class.getName())
.build();
if (session == null || session.isClosed()) {
var host = System.getenv("CASSANDRA_HOST") == null ? "localhost" : System.getenv("CASSANDRA_HOST");
var username = "localhost".equals(host)? "": "cassandra";
var password = "localhost".equals(host)? "": "cassandra";
LOG.info("Cassandra host '{}'.", host);
LOG.info("Cassandra username '{}'.", username);
LOG.info("Cassandra password '{}'.", password);
var sessionBuilder = new CqlSessionBuilder()
.addContactPoint(new InetSocketAddress(host, 9042))
.withLocalDatacenter("datacenter1")
.withConfigLoader(loader);
if (!username.isEmpty()) {
sessionBuilder.withAuthCredentials(username, password);
}
session = sessionBuilder.build();
It also important to mention that I have lots 170+ test cases distributed on different files, and with every file execution, am trying to clean and populate the DB again using this code
session.execute("create keyspace if not exists \"schema_x\" WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 1};");
for (final String stmt : getCassaandraStatementsFromFile(CASSANDRA_SCHEMA_FILE)) {
session.execute(stmt);
LOG.info("Cassandra. Executed statement: '{}'.", stmt.replaceAll("\n", ""));
}
and the error exactly happening on that create keyspace line, I tried to apply some tuning from my side by
adapting the connection loader and using throttling but it didn't help
I also checked the local-datacenter value inside docker itself and matched mine.
Finally the complete error stacktrace in case it is required
org.springframework.data.cassandra.CassandraConnectionFailureException: \
Query; CQL [com.datastax.oss.driver.internal.core.cql.DefaultSimpleStatement#65b70f9e]; \
No node was available to execute the query; nested exception is \
com.datastax.oss.driver.api.core.NoNodeAvailableException: \
No node was available to execute the query
at org.springframework.data.cassandra.core.cql.CassandraExceptionTranslator.translate(CassandraExceptionTranslator.java:137)
at org.springframework.data.cassandra.core.cql.CassandraAccessor.translate(CassandraAccessor.java:422)
at org.springframework.data.cassandra.core.cql.CqlTemplate.translateException(CqlTemplate.java:764)
at org.springframework.data.cassandra.core.cql.CqlTemplate.query(CqlTemplate.java:300)
at org.springframework.data.cassandra.core.cql.CqlTemplate.query(CqlTemplate.java:320)
at org.springframework.data.cassandra.core.CassandraTemplate.select(CassandraTemplate.java:337)
at org.springframework.data.cassandra.repository.query.CassandraQueryExecution$CollectionExecution.execute(CassandraQueryExecution.java:136)
at
...
Caused by: com.datastax.oss.driver.api.core.NoNodeAvailableException: \
No node was available to execute the query
at com.datastax.oss.driver.api.core.NoNodeAvailableException.copy(NoNodeAvailableException.java:40)
at com.datastax.oss.driver.internal.core.util.concurrent.CompletableFutures.getUninterruptibly(CompletableFutures.java:149)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:53)
at com.datastax.oss.driver.internal.core.cql.CqlRequestSyncProcessor.process(CqlRequestSyncProcessor.java:30)
at com.datastax.oss.driver.internal.core.session.DefaultSession.execute(DefaultSession.java:230)
at com.datastax.oss.driver.api.core.cql.SyncCqlSession.execute(SyncCqlSession.java:54)
at org.springframework.data.cassandra.core.cql.CqlTemplate.query(CqlTemplate.java:298)
... 39 common frames omitted
I would appreciate your support on this, and thanks in advance

Answer 1

Spring uses the Cassandra Java driver to connect to Cassandra clusters.
For each query execution, the Java driver generates a query plan which contains alist of nodes to connect to to execute a query. The configured load-balancing policy (DcInferringLoadBalancingPolicy in your case) determines the nodes to be included in the query plan. The query plan will only contain nodes which are known to be available which means the policy will not include nodes which are known to be down or ignored (see Load balancing with the Java driver for details).
In a scenario where ALL the nodes have been marked "down" or "ignored", the driver has no choice since there are no nodes available to connect to so the driver throws NoNodeAvailableException. As the error message you posted above states, there is literally no node available to execute the query -- exactly as it says.
The driver marks nodes as "down" or "ignored" because they haven't responded in some time usually because they are overloaded. Consider throttling the load even further so nodes are not overloaded.
Additionally, schema changes do not follow the same path as regular writes (INSERT, UPDATE, DELETE). Each DDL change (CREATE, ALTER, DROP) is propagated to other nodes via the gossip protocol so depending on the size of the cluster, it can take some time for all nodes in the cluster to reach schema agreement.
When performing schema changes programmatically, don't fire off changes in quick succession or you will risk nodes getting out of sync. Your application should pause after each schema change and check that all nodes has reached schema agreement BEFORE executing the next schema change, for example with a call to isSchemaInAgreement() or asynchronously with checkSchemaAgreementAsync(). For details, see Schema agreement with the Java driver.
As a side note, the default cassandra superuser is not designed for general use. It should only ever be used to provision another superuser account then deleted.
The use of the default cassandra superuser account is expensive since it requires a QUORUM of nodes to authenticate. In contrast, all other accounts other than cassandra authenticate with a consistency of ONE. Cheers!
👉 Please support the Apache Cassandra community by hovering over the cassandra tag then click on the Watch tag button. 🙏 Thanks!