Materialize index throws DB::Exception: Missing columns - clickhouse

Question

I met a problem for materialize index with version 21.3.
I create a table with 3 coulmns u64, i32 and s.
CREATE TABLE test_idx (`u64` UInt64, `i32` Int32, `s` String) ENGINE = MergeTree() ORDER BY u64;
Then I insert 3 millions data in it and run OPTIMIZE TABLE test_idx FINAL(to make part Wide)
select name, part_type from system.parts where table='test_idx' and active=1;
┌─name──────────────────┬─part_type─┐
│ all_1_21762_111_21773 │ Wide │
└───────────────────────┴───────────┘
Then I add tow indexes to the table
alter table test_idx add INDEX a (u64, s) TYPE minmax GRANULARITY 3;
alter table test_idx add INDEX b (i32 * length(s)) TYPE set(1000) GRANULARITY 4;
Then I materialize the index a to make the index useful for old data.
alter table test_idx materialize index a;
Here's the Exception and stack trace.
2022.07.14 04:06:38.192403 [ 11633 ] {} <Error> DB::IBackgroundJobExecutor::jobExecutingTask()::<lambda()>: Code: 47, e.displayText() = DB::Exception: Missing columns: 'i32' while processing query: 'u64, s, i32 * length(s)', required columns: 'u64' 's' 'i32' 'u64' 's' 'i32', Stack trace (when copying this message, always include the lines below):
0. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) # 0xe16fb61 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
1. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../src/Common/Exception.cpp:55: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) # 0x476f358 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
2. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../src/Interpreters/TreeRewriter.cpp:752: DB::TreeRewriterResult::collectUsedColumns(std::__1::shared_ptr<DB::IAST> const&, bool) (.cold) # 0x416fba2 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
3. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/new:237: DB::TreeRewriter::analyze(std::__1::shared_ptr<DB::IAST>&, DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::IStorage const>, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, bool, bool) const # 0xa8a1228 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
4. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/list:753: DB::MergeTreeDataMergerMutator::getIndicesToRecalculate(std::__1::shared_ptr<DB::IBlockInputStream>&, DB::NamesAndTypesList const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::Context const&) # 0xac80bbb in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
5. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/list:753: DB::MergeTreeDataMergerMutator::mutatePartToTemporaryPart(DB::FutureMergedMutatedPart const&, std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::MutationCommands const&, DB::BackgroundProcessListEntry<DB::MergeListElement, DB::MergeInfo>&, long, DB::Context const&, std::__1::unique_ptr<DB::IReservation, std::__1::default_delete<DB::IReservation> > const&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) # 0xac87d74 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
6. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/type_traits:3934: DB::StorageMergeTree::mutateSelectedPart(std::__1::shared_ptr<DB::StorageInMemoryMetadata const> const&, DB::StorageMergeTree::MergeMutateSelectedEntry&, std::__1::shared_ptr<DB::RWLockImpl::LockHolderImpl>&) # 0xaaf12d2 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
7. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../src/Storages/StorageMergeTree.cpp:967: bool std::__1::__function::__policy_invoker<bool ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::StorageMergeTree::getDataProcessingJob()::'lambda'(), bool ()> >(std::__1::__function::__policy_storage const*) # 0xaaf15dc in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
8. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../src/Storages/MergeTree/BackgroundJobsExecutor.cpp:103: void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::IBackgroundJobExecutor::jobExecutingTask()::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) # 0xabefd13 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
9. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/functional:2212: ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) # 0x47cd3a2 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
10. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../src/Common/ThreadPool.h:181: ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()>(void&&, void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()&&...)::'lambda'()::operator()() # 0x47cd84e in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
11. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/functional:2212: ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) # 0x47ccb42 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
12. /root/zyf/workspace/clickhouse/build/RelWithDebInfo/../../contrib/libcxx/include/memory:1655: void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda1'()> >(void*) # 0x47cb4f3 in /root/zyf/workspace/clickhouse/build/RelWithDebInfo/bin/clickhouse
13. start_thread # 0x8609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
14. __clone # 0x11f133 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
(version 21.3.14.1.7)
Column i32 is not used for index a, but why here shows Missing columns? Or I can't use skipping index in this way?

Answer 1

Upgrading to the latest version of ClickHouse should resolve your issue. Is there any specific reason for you to use 21.3?

Answer 2

Try detach all the data parts that causes the error (assuming that the MATERIALIZE mutation had succeeded on most data parts), and attach them to a new table with same structure, do MATERIALIZE on the new table, and hopefully it will complete with no error. Finally ALTER TABLE ... MOVE PARTITION ... to move all partitions back to the old table.