Slow counts finally solved - phoenix-framework

Question

tl;dr: |> Repo.aggregate(:count, :id) is slow, do use |> Repo.aggregate(:count)
I'm running a podcast database with > 5 million episodes. After storing a new episode, I count the episodes for a given podcast for a counter cache like this:
episodes_count = where(Episode, podcast_id: ^podcast_id)
|> Repo.aggregate(:count, :id)
This turned out to get slower and slower. So I started to dig deeper, and I realized, that
in Postgres 12 only SELECT COUNT(*) does an index only scan, while SELECT COUNT(e0.id) doesn't.
For a cold database (just restarted) even the first index scan is reasonably fast:
postgres=# \c pan_prod
Sie sind jetzt verbunden mit der Datenbank »pan_prod« als Benutzer »postgres«.
pan_prod=# EXPLAIN ANALYZE SELECT count(*) FROM "episodes" AS e0 WHERE (e0."podcast_id" = 35202);
QUERY PLAN
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=348.51..348.52 rows=1 width=8) (actual time=15.823..15.823 rows=1 loops=1)
-> Index Only Scan using episodes_podcast_id_index on episodes e0 (cost=0.43..323.00 rows=10204 width=0) (actual time=1.331..14.832 rows=10613 loops=1)
Index Cond: (podcast_id = 35202)
Heap Fetches: 0
Planning Time: 2.994 ms
Execution Time: 16.017 ms
It even gets faster for the second scan:
pan_prod=# EXPLAIN ANALYZE SELECT count(*) FROM "episodes" AS e0 WHERE (e0."podcast_id" = 35202);
QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=348.51..348.52 rows=1 width=8) (actual time=5.007..5.008 rows=1 loops=1)
-> Index Only Scan using episodes_podcast_id_index on episodes e0 (cost=0.43..323.00 rows=10204 width=0) (actual time=0.042..3.548 rows=10613 loops=1)
Index Cond: (podcast_id = 35202)
Heap Fetches: 0
Planning Time: 0.304 ms
Execution Time: 5.074 ms
While the first bitmap heap scan ist terribly slow:
pan_prod=# EXPLAIN ANALYZE SELECT count(e0.id) FROM "episodes" AS e0 WHERE (e0."podcast_id" = 35202);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=37181.71..37181.72 rows=1 width=8) (actual time=4098.525..4098.526 rows=1 loops=1)
-> Bitmap Heap Scan on episodes e0 (cost=219.51..37156.20 rows=10204 width=4) (actual time=6.508..4082.558 rows=10613 loops=1)
Recheck Cond: (podcast_id = 35202)
Heap Blocks: exact=6516
-> Bitmap Index Scan on episodes_podcast_id_index (cost=0.00..216.96 rows=10204 width=0) (actual time=3.657..3.658 rows=10613 loops=1)
Index Cond: (podcast_id = 35202)
Planning Time: 0.412 ms
Execution Time: 4098.719 ms
The second one is typically faster:
pan_prod=# EXPLAIN ANALYZE SELECT count(e0.id) FROM "episodes" AS e0 WHERE (e0."podcast_id" = 35202);
QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------
Aggregate (cost=37181.71..37181.72 rows=1 width=8) (actual time=18.857..18.857 rows=1 loops=1)
-> Bitmap Heap Scan on episodes e0 (cost=219.51..37156.20 rows=10204 width=4) (actual time=6.047..17.152 rows=10613 loops=1)
Recheck Cond: (podcast_id = 35202)
Heap Blocks: exact=6516
-> Bitmap Index Scan on episodes_podcast_id_index (cost=0.00..216.96 rows=10204 width=0) (actual time=3.738..3.738 rows=10613 loops=1)
Index Cond: (podcast_id = 35202)
Planning Time: 0.322 ms
Execution Time: 18.999 ms
I don't get, why SELECT count(e0.id) doesn't use the index, and I would like to know why.
I always thought, I should prefer it, as only one column is looked at, but that is not the case, it seems.

Answer 1

I don't get, why SELECT count(e0.id) doesn't use the index, and I would like to know why. I always thought, I should prefer it, as only one column is looked at, but that is not the case, it seems.
It does use the index episodes_podcast_id_index both times.
Once it can do without additionally looking in the table, the other time it does look into the table.
You didn't provide the index defintion, but it seems like id is not part of it. I would also supect that it is not a not nullable column (it might contains null values). Thus, the DB has to fetch the id column to see if it is null.
Why?
Because count(*) counts rows while count(<expression>) counts not-null values.
See: https://modern-sql.com/concept/null#aggregates