count() query on ClickHouse replication - clickhouse

Question

Hello ClickHouse expects,
I am testing ClickHouse replications without zookeeper to understand how it works and have 2 questions.
How I tested:
Setup 3 clickhouse-server (CH1, CH2, CH3) on different VMs (21.9.4 revision 54449)
Have a cluster using these 3 servers (see below for config)
Have a MergeTree (log_local) and a Distributed (log_all) tables
Send 100M logs to one server (CH1) through log_all; using clickhouse-client which is in a different VM
(Q1) After the insertion, I query count() of log_local from all 3 servers and the total number is as expected (i.e. 200M). However, when I query using log_all, the outcomes are different among servers (close to 200M but not exact). And even stranger, the count changes even within the same server. Can you please explain this behavior? Can it be a configuration issue? With no replica (3shards_1replica) test, I don’t see this count difference.
I see this is not a recommended so, eventually, I’d use a cluster coordinator - hoping clickhouse-keeper is in production by then. Before that stage, I am assessing if I can use this as a temporary solution with explainable shortcomings.
(Q2) This is more a generic question on replication. The count of log_all is 200M which include replica’s. What is the practical way that I query it without replica? I.e., select count() from log_all (or a different name) yields 100M not 200M.
Configs (I have modified some names from the original for not showing private information):
# remote_servers
<log_3shards_2replicas>
<shard>
<replica>
<host>CH1</host>
<port>9000</port>
</replica>
<replica>
<host>CH2</host>
<port>9000</port>
</replica>
</shard>
<shard>
<replica>
<host>CH2</host>
<port>9000</port>
</replica>
<replica>
<host>CH3</host>
<port>9000</port>
</replica>
</shard>
<shard>
<replica>
<host>CH3</host>
<port>9000</port>
</replica>
<replica>
<host>CH1</host>
<port>9000</port>
</replica>
</shard>
</log_3shards_2replicas>
ENGINE = MergeTree
PARTITION BY toYYYYMMDD(my_time)
ORDER BY my_time
SETTINGS index_granularity = 8192
ENGINE = Distributed(‘log_3shards_2replicas', ‘my_db’, ‘log_local', rand())
Some references:
https://github.com/ClickHouse/ClickHouse/issues/2161
fillimonov commented - “Actually you can create replication without Zookeeper and ReplicatedMergeTree, just by using Distributed table above MergeTree and internal_replication=false cluster setting, but in that case there will no guarantee that all the replicas will have 100% the same data, so i rather would not recommend that scenario.”
Similar issue discussions:
https://github.com/ClickHouse/ClickHouse/issues/1443
https://github.com/ClickHouse/ClickHouse/issues/6735
Thanks in advance.

Answer 1

You have configured a "circle-replication". God help you.
THIS CONFIGURATION IS NOT SUPPORTED and is not covered by tests in CI.
Circle-replication is hard to configure, very unobvious for newbies.
Circle-replication provides a lot of issues and hardiness to debug.
A lot queries yield incorrect results.
The most of users who used circle-replication in the past moved to usual setup N shards * M replica and happy now.
https://kb.altinity.com/engines/
https://youtu.be/4DlQ6sVKQaA
In your configuration DEFAULT_DATABASE attribute is missing, circle-replication unable to work without it.