Cassandra READ operations latency dependency of WRITE operations - performance

Question

I'm testing Cassandra performance in the case of simultaneous read and writes operations.
Latency for READ operations is important for me, WRITES usually are done once per day, so I'm investigating if the WRITE operations would affect READ operations latency.
I have a table of 10_000 records which I READ from application at 50 RPS. For example, the latency is 10ms, REQUEST_SERIAL_CONSISTENCY and REQUEST_CONSISTENCY is QUORUM for READ operations.
Query for READs is: SELECT * from table where id = X
Simultaneously I run another application which insert same records to table (with same ids, so in fact it is UPDATE operations), the REQUEST_SERIAL_CONSISTENCY and REQUEST_CONSISTENCY is ALL for WRITE operations. In several threads, I load my Cassandra cluster with batches of INSERT queries size of 60 queries in batch.
Query for WRITEs is: INSERT into table values(id,..) where id = X;
What I expect it is the latency increasing for READ operations when WRITES is enabled. But latency is stable in general besides random spikes like:
Why hasn't latency been changed, was it wrong to expect uniform latency increase?
What about these spikes, can it be connected with MemTable flushes to disk? I can see them in statistic:
nodetool tpstats
MemtableFlushWriter 0 0 4272 0 0
Info of my cluster:
cqlsh> SELECT * FROM system_schema.keyspaces;
keyspace_name | durable_writes | replication
--------------------+----------------+-------------------------------------------------------------------------------------
system_auth | True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
system_schema | True | {'class': 'org.apache.cassandra.locator.LocalStrategy'}
system_distributed | True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '3'}
system | True | {'class': 'org.apache.cassandra.locator.LocalStrategy'}
search | True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '1'}
system_traces | True | {'class': 'org.apache.cassandra.locator.SimpleStrategy', 'replication_factor': '2'}

Answer 1

With 10000 records, I do not expect much change in latency. Latency can get hit due to lot of factor. It could be that Cassandra is doing I/O for compactions, repairs and that could hit your latency even when no write is done. So I assume random spikes you are seeing is because of some compactions running in background. One more thing, when you say you are inserting records in batch of 60, please dont use Cassandra batch for inserting records as they will be slow.