Upgrade ClickHouse from 1.1.54231 to 19.6.2.11 - clickhouse

Question

We finally decided to update our production server ClickHouse version 1.1.54231 to a new 19.6 . The new version started, picked up all the configs, sees the data on the server too, but cannot pick up new data on Zookeeper. In a log errors
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: user_files_path (version 19.6.2.11 (official build))
Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = Not found: format_schema_path (version 19.6.2.11 (official build))
Include not found: clickhouse_remote_servers
Include not found: clickhouse_compression
Logging error to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging error to console
2019.05.16 17:06:10.801816 [ 1 ] {} <Error> Application: Listen [0.0.0.0]: 98: Net Exception: Address already in use: 0.0.0.0:8123 If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2019.05.16 17:06:10.859589 [ 25 ] {} <Error> analytics..inner.main_page_hits (StorageReplicatedMergeTree): Code: 40, e.displayText() = DB::Exception: Checksums of parts don't match: hash of uncompressed files doesn't match (version 19.6.2.11 (official build)). Data after merge is not byte-identical to data on another replicas. There could be several reasons:
1. Using newer version of compression library after server update.
2. Using another compression method.
3. Non-deterministic compression algorithm (highly unlikely).
4. Non-deterministic merge algorithm due to logical error in code.
5. Data corruption in memory due to bug in code.
6. Data corruption in memory due to hardware issue.
7. Manual modification of source data after server startup.
8. Manual modification of checksums stored in ZooKeeper. We will download merged part from replica to force byte-identical result.
2019.05.16 17:06:10.874886 [ 23 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8., Stack trace:
0. /usr/bin/clickhouse-server(StackTrace::StackTrace()+0x16) [0x767c3f6]
1. /usr/bin/clickhouse-server(DB::Exception::Exception(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x22) [0x39294a2]
2. /usr/bin/clickhouse-server() [0x6d7fc8c]
3. /usr/bin/clickhouse-server(DB::DataPartsExchange::Fetcher::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, DB::ConnectionTimeouts const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x950) [0x6d830b0]
4. /usr/bin/clickhouse-server() [0x6d2d7a4]
5. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::fetchPart(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, unsigned long)+0x97f) [0x6d48d5f]
6. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeFetch(DB::ReplicatedMergeTreeLogEntry&)+0x7f2) [0x6d4b012]
7. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::executeLogEntry(DB::ReplicatedMergeTreeLogEntry&)+0x26b) [0x6d4bd5b]
8. /usr/bin/clickhouse-server() [0x6d4c31f]
9. /usr/bin/clickhouse-server(DB::ReplicatedMergeTreeQueue::processEntry(std::function<std::shared_ptr<zkutil::ZooKeeper> ()>, std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&, std::function<bool (std::shared_ptr<DB::ReplicatedMergeTreeLogEntry>&)>)+0x64) [0x6e6d324]
10. /usr/bin/clickhouse-server(DB::StorageReplicatedMergeTree::queueTask()+0x189) [0x6d280d9]
11. /usr/bin/clickhouse-server(DB::BackgroundProcessingPool::threadFunction()+0x44e) [0x6d7da4e]
12. /usr/bin/clickhouse-server() [0x6d7e4c9]
13. /usr/bin/clickhouse-server(ThreadPoolImpl<std::thread>::worker(std::_List_iterator<std::thread>)+0x1ab) [0x768238b]
14. /usr/bin/clickhouse-server() [0xb5488bf]
15. /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f553fd076db]
16. /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f553f28688f]
(version 19.6.2.11 (official build))
2019.05.16 17:06:10.886495 [ 25 ] {} <Error> analytics.actions (StorageReplicatedMergeTree): DB::StorageReplicatedMergeTree::queueTask()::<lambda(DB::StorageReplicatedMergeTree::LogEntryPtr&)>: Code: 33, e.displayText() = DB::Exception: Cannot read all data. Bytes read: 0. Bytes expected: 8.,
Any ideas?

Answer 1

It looks like you have updated only one replica out of few.
In replicated scenarios background merge results are crosschecked using checksums. If the result of merge on 2 replicas is not byte identical replica send that message to logs and redownload the problematic part from other replicas. After you will update other replicas problem will disappear.