Clickhouse table stuck in read-only mode - clickhouse

Question

I am new to clickhouse
So on one my system i am seeing this issue repeatedly:
"{} \u003cError\u003e void DB::AsynchronousMetrics::update(): Cannot
get replica delay for table: people: Code: 242, e.displayText() =
DB::Exception: Table is in readonly mode, Stack trace:"
And i can see my zookeeper was not in good state, So from the clickhouse docs, it seems this is related to either
Metadata in zookeeper got deleted somehow
Zookeeper was not up when clickhouse was trying to comes up.
Either way i want to recover from the error and docs suggested below steps:
a)To start recovery, create the node /path_to_table/replica_name/flags/force_restore_data in ZooKeeper
with any content, or run the command to restore all replicated tables:
sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data
Then restart the server. On start, the server deletes these flags and starts recovery.
But i am not able to understand where should i run this command, i looked inside the clickhouse container under the location /var/lib/clickhouse there is not flags directory. Should i create it first??
Also is there a way to recover from this error without restarting the server, i would rather avoid container restart??
Attaching few relevant logs before the read only exception:
2020.06.19 16:49:02.789216 [ 13 ] {} <Error> DB_0.people (ReplicatedMergeTreeRestartingThread): Couldn't start replication: Replica /clickhouse/tables/shard_0/people/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/people/replicas/replica_0732646014/is_active manually, DB::Exception: Replica /clickhouse/tables/shard_0/people/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/people/replicas/replica_0732646014/is_active manually, stack trace:
2020.06.19 16:49:13.576855 [ 17 ] {} <Error> DB_0.school (ReplicatedMergeTreeRestartingThread): Couldn't start replication: Replica /clickhouse/tables/shard_0/school/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/school/replicas/replica_0732646014/is_active manually, DB::Exception: Replica /clickhouse/tables/shard_0/school/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/school/replicas/replica_0732646014/is_active manually, stack trace:
2020.06.19 16:49:23.497824 [ 19 ] {} <Error> DB_0.people (ReplicatedMergeTreeRestartingThread): Couldn't start replication: Replica /clickhouse/tables/shard_0/people/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/people/replicas/replica_0732646014/is_active manually, DB::Exception: Replica /clickhouse/tables/shard_0/people/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/people/replicas/replica_0732646014/is_active manually, stack trace:
2020.06.19 16:49:23.665089 [ 20 ] {} <Error> DB_0.school (ReplicatedMergeTreeRestartingThread): Couldn't start replication: Replica /clickhouse/tables/shard_0/school/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/school/replicas/replica_0732646014/is_active manually, DB::Exception: Replica /clickhouse/tables/shard_0/school/replicas/replica_0732646014 appears to be already active. If you're sure it's not, try again in a minute or remove znode /clickhouse/tables/shard_0/school/replicas/replica_0732646014/is_active manually, stack trace:
2020.06.19 16:49:59.703591 [ 41 ] {} <Error> void Coordination::ZooKeeper::receiveThread(): Code: 999, e.displayText() = Coordination::Exception: Operation timeout (no response) for path: /clickhouse/tables/shard_0/school/blocks (Operation timeout), Stack trace:
2020.06.19 16:49:59.847751 [ 18 ] {} <Error> DB_0.people: void DB::StorageReplicatedMergeTree::queueUpdatingTask(): Code: 999, e.displayText() = Coordination::Exception: Connection loss, path: /clickhouse/tables/shard_0/people/mutations, Stack trace:
2020.06.19 16:50:00.205911 [ 19 ] {} <Warning> DB_0.school (ReplicatedMergeTreeRestartingThread): ZooKeeper session has expired. Switching to a new session.
2020.06.19 16:50:00.315063 [ 19 ] {} <Error> zkutil::EphemeralNodeHolder::~EphemeralNodeHolder(): Code: 999, e.displayText() = Coordination::Exception: Session expired (Session expired), Stack trace:
2020.06.19 16:50:00.338176 [ 15 ] {} <Error> DB_0.people: void DB::StorageReplicatedMergeTree::mergeSelectingTask(): Code: 999, e.displayText() = Coordination::Exception: Session expired (Session expired), Stack trace:
2020.06.19 16:50:00.387589 [ 16 ] {} <Error> DB_0.school: void DB::StorageReplicatedMergeTree::mergeSelectingTask(): Code: 999, e.displayText() = Coordination::Exception: Connection loss, path: /clickhouse/tables/shard_0/school/log, Stack trace:
2020.06.19 16:50:00.512689 [ 17 ] {} <Error> zkutil::EphemeralNodeHolder::~EphemeralNodeHolder(): Code: 999, e.displayText() = Coordination::Exception: Session expired (Session expired), Stack trace:
2020.06.19 16:50:20.753596 [ 47 ] {} <Error> void DB::DDLWorker::runMainThread(): Code: 999, e.displayText() = Coordination::Exception: All connection tries failed while connecting to ZooKeeper. Addresses: 172.16.0.28:2181
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Timeout: connect timed out: 172.16.0.28:2181 (version 19.13.1.11 (official build)), 172.16.0.28:2181
Code: 209, e.displayText() = DB::NetException: Timeout exceeded while reading from socket (172.16.0.28:2181): while receiving handshake from ZooKeeper (version 19.13.1.11 (official build)), 172.16.0.28:2181
Code: 209, e.displayText() = DB::NetException: Timeout exceeded while reading from socket (172.16.0.28:2181): while receiving handshake from ZooKeeper (version 19.13.1.11 (official build)), 172.16.0.28:2181
(Connection loss), Stack trace:
2020.06.19 16:50:31.499775 [ 51 ] {} <Error> void DB::AsynchronousMetrics::update(): Cannot get replica delay for table: DB_0.people: Code: 242, e.displayText() = DB::Exception: Table is in readonly mode, Stack trace:
Edit: I did manage to find the folder where flags is present(its present in my volume /repo/data) but when i try to run the command
sudo -u clickhouse touch /repo/data/flags/force_restore_data
I got this:
Use one of the following commands:
clickhouse local [args]
clickhouse client [args]
clickhouse benchmark [args]
clickhouse server [args]
clickhouse extract-from-config [args]
clickhouse compressor [args]
clickhouse format [args]
clickhouse copier [args]
clickhouse obfuscator [args]