Cassandra DateTieredCompactionStrategy SSTables - performance

Question

We are running Cassandra 2.2.3.
Have a column family using DateTieredCompactionStrategy as follows,
CREATE TABLE test (
num_id text,
position_time timestamp,
acc int,
coordinate text,
device_no text,
PRIMARY KEY (num_id, position_time, coordinate)
) WITH CLUSTERING ORDER BY (position_time DESC, coordinate ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = 'table for gps points from car gps source.'
AND compaction = {'timestamp_resolution': 'MILLISECONDS', 'max_sstable_age_days': '8', 'base_time_seconds': '3600', 'class': 'org.apache.cassandra.db.compaction.DateTieredCompactionStrategy'}
AND compression = {'chunk_length_kb': '64', 'crc_check_chance': '1.0', 'sstable_compression': 'org.apache.cassandra.io.compress.SnappyCompressor'}
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 86400
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99.0PERCENTILE';
We do have some traffic, It keeps inserting the table.
Cassandra generates many SStables, around 2,000 in total.
For example,
-rw-r--r-- 1 cassandra cassandra 86M Jan 20 02:59 la-11110-big-Data.db
-rw-r--r-- 1 cassandra cassandra 111M Jan 20 03:11 la-11124-big-Data.db
-rw-r--r-- 1 cassandra cassandra 176M Jan 20 03:12 la-11125-big-Data.db
-rw-r--r-- 1 cassandra cassandra 104M Jan 20 03:14 la-11130-big-Data.db
-rw-r--r-- 1 cassandra cassandra 102M Jan 20 03:26 la-11144-big-Data.db
-rw-r--r-- 1 cassandra cassandra 172M Jan 20 03:26 la-11145-big-Data.db
-rw-r--r-- 1 cassandra cassandra 107M Jan 20 03:30 la-11149-big-Data.db
-rw-r--r-- 1 cassandra cassandra 96M Jan 20 03:41 la-11163-big-Data.db
-rw-r--r-- 1 cassandra cassandra 176M Jan 20 03:41 la-11164-big-Data.db
-rw-r--r-- 1 cassandra cassandra 97M Jan 20 03:45 la-11169-big-Data.db
-rw-r--r-- 1 cassandra cassandra 82M Jan 20 03:57 la-11183-big-Data.db
-rw-r--r-- 1 cassandra cassandra 194M Jan 20 03:58 la-11184-big-Data.db
-rw-r--r-- 1 cassandra cassandra 28M Jan 20 03:59 la-11187-big-Data.db
-rw-r--r-- 1 cassandra cassandra 90M Jan 20 04:00 la-11188-big-Data.db
My question is, Is it normal to have so many SStables (2000)?
The other thing is we are experiencing readtimeout Exception for selection query.
The selection query uses primary key num_id and clustering key timestamp.
The readtimeout is set to 10 seconds.
So, the other question is the readtimeout exception is caused by many SStables or wide row? How to solve avoid this exception?

Answer 1

the problem is "'timestamp_resolution': 'MILLISECONDS'" I've filed https://issues.apache.org/jira/browse/CASSANDRA-11041 to improve documentation about this parameter

Answer 2

My question is, Is it normal to have so many SStables (2000)?
No, it's not normal. I think that in your case, the compaction is not fast enough to keep up with the ingestion rate. What kind of hard drive do you have for the Cassandra server ? Spinning disk ? SSD ? Shared storage ?
So, the other question is the readtimeout exception is caused by many SStables or wide row?
It can be both, but in your case, I'm pretty sure that it's related to the huge number of SSTables
How to solve avoid this exception?
Check that your disk I/O can keep up. Use dstat and iostat Linux tool to monitor the I/O