.sst files are daily increasing in cockroachdb on path /var/lib/cockroach/cockroach-data - cockroachdb

Question

.sst files are daily increasing in cockroachdb on path /var/lib/cockroach/cockroach-data
daily there are lots of .sst files are getting created in my cockroachdb cluster at the host path /var/lib/cockroach/cockroach-data.
Can I delete them? its a critical environment and We can not just delete those files without knowing the impact.
however, they are piling up the disk utilization.
[root#winprd-vck001 cockroach-data]# ls -lrth | grep sst | wc -l
4616
-rw-r----- 1 cockroach cockroach 8.9M May 10 21:27 5266230.sst
-rw-r----- 1 cockroach cockroach 50M May 10 21:35 5266274.sst
-rw-r----- 1 cockroach cockroach 17M May 10 22:00 5266420.sst
-rw-r----- 1 cockroach cockroach 12M May 10 22:00 5266421.sst
-rw-r----- 1 cockroach cockroach 17M May 10 22:11 5266468.sst
-rw-r----- 1 cockroach cockroach 9.0M May 10 23:04 5266745.sst
-rw-r----- 1 cockroach cockroach 17M May 10 23:04 5266746.sst
-rw-r----- 1 cockroach cockroach 18M May 10 23:19 5266806.sst
-rw-r----- 1 cockroach cockroach 31M May 10 23:19 5266807.sst
-rw-r----- 1 cockroach cockroach 11M May 11 00:11 5267051.sst
-rw-r----- 1 cockroach cockroach 28M May 11 00:15 5267083.sst
-rw-r----- 1 cockroach cockroach 64M May 11 00:15 5267089.sst
-rw-r----- 1 cockroach cockroach 20M May 11 00:30 5267173.sst
-rw-r----- 1 cockroach cockroach 1.2M May 11 00:30 5267174.sst
-rw-r----- 1 cockroach cockroach 64M May 11 01:01 5267295.sst
-rw-r----- 1 cockroach cockroach 64M May 11 01:01 5267296.sst
-rw-r----- 1 cockroach cockroach 64M May 11 01:01 5267297.sst
-rw-r----- 1 cockroach cockroach 14M May 11 01:01 5267298.sst
-rw-r----- 1 cockroach cockroach 17M May 11 01:20 5267403.sst
-rw-r----- 1 cockroach cockroach 18M May 11 01:20 5267404.sst
-rw-r----- 1 cockroach cockroach 571K May 11 01:20 5267405.sst
-rw-r----- 1 cockroach cockroach 9.6M May 11 01:57 5267591.sst
-rw-r----- 1 cockroach cockroach 17M May 11 01:57 5267592.sst
-rw-r----- 1 cockroach cockroach 12M May 11 01:57 5267593.sst
-rw-r----- 1 cockroach cockroach 65M May 11 01:58 5267646.sst
-rw-r----- 1 cockroach cockroach 19M May 11 01:58 5267647.sst
-rw-r----- 1 cockroach cockroach 65M May 11 01:58 5267674.sst
-rw-r----- 1 cockroach cockroach 13M May 11 01:58 5267680.sst
-rw-r----- 1 cockroach cockroach 64M May 11 02:33 5267929.sst
-rw-r----- 1 cockroach cockroach 19M May 11 02:33 5267930.sst
-rw-r----- 1 cockroach cockroach 27M May 11 03:44 5268110.sst
-rw-r----- 1 cockroach cockroach 4.5M May 11 03:51 5268155.sst
-rw-r----- 1 cockroach cockroach 34M May 11 03:51 5268156.sst
-rw-r----- 1 cockroach cockroach 2.6M May 11 04:14 5268235.sst
-rw-r----- 1 cockroach cockroach 8.3M May 11 05:17 5268436.sst
-rw-r----- 1 cockroach cockroach 17M May 11 06:00 5268696.sst

Answer 1

Those .sst files are your data, don't delete them.
If your disk utilization is higher than you'd like, but in a relatively steady state after 10 days of uptime, try decreasing your gc.ttlseconds zone configuration so that deleted rows are garbage-collected more frequently. Any time that is comfortably longer than your slowest query is generally safe, although you'll want to make sure not to interfere with backups or change data capture jobs.
If your disk utilization is continuing to creep up over time, either
Your data is growing faster than you're deleting rows: This is a good problem to have. You can solve it with money by adding more nodes, or by periodically deleting stale data (or archiving it in an external system). In CockroachDB 22.1 (in beta release as of May 11th 2022), you can automate the first part of this process by setting ttl_expire_after on a table in a CREATE or ALTER statement. Archiving can be done via scheduled backups or changefeeds.
Something is preventing garbage collection: Backups and changefeeds will protect old versions of rows from being garbage-collected until they've finished exporting them. If you have a long-running job that's not able to connect to its external sink, it could be preventing garbage collection indefinitely. This can be fixed by fixing the job. Other long-running jobs can prevent garbage collection of metadata if something's gone wrong.
Finally, if you're still in the first 10 days of a new cluster and you're seeing disk utilization slowly rise, that's expected. Your database is growing, and it's also accumulating time series statistical data for use in monitoring. No need to take action just yet.