Elasticsearch allocates primary shards only on first node in the cluster - elasticsearch

Question

I have 5 node elasticsearch cluster to store logs ( 4 master eligble and data and 1 master only node)
and all primary shards that was created allocate only on firt node in the cluster so the 1st node almost totally overloaded by cpu
why this happens? nodes have the same specs (cpu,ram,etc)
here is my cat/allocation ( i manually move shards from elastic-01 every morning, and on elastic-01 is fewer shards that on the others, but anyway load on elastic-01 is big)
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
354 4.9tb 4.9tb 804.7gb 5.7tb 86 10.0.5.22 10.0.5.252 elastic-01
435 4tb 4tb 1.7tb 5.7tb 70 10.0.5.23 10.0.5.23 elastic-02
434 4.6tb 4.6tb 1tb 5.7tb 80 10.0.5.27 10.0.5.27 elastic-06
434 4.7tb 4.7tb 1014.8gb 5.7tb 82 10.0.5.28 10.0.5.28 elastic-07
and here is nodeattrs
node host ip attr value
elastic-07 10.0.5.28 10.0.5.28 ml.machine_memory 42002579456
elastic-07 10.0.5.28 10.0.5.28 ml.max_open_jobs 512
elastic-07 10.0.5.28 10.0.5.28 xpack.installed true
elastic-07 10.0.5.28 10.0.5.28 ml.max_jvm_size 27917287424
elastic-07 10.0.5.28 10.0.5.28 zone zone2
elastic-07 10.0.5.28 10.0.5.28 transform.node true
elastic-01 10.0.5.22 10.0.5.22 ml.machine_memory 42002583552
elastic-01 10.0.5.22 10.0.5.22 ml.max_open_jobs 512
elastic-01 10.0.5.22 10.0.5.22 xpack.installed true
elastic-01 10.0.5.22 10.0.5.22 ml.max_jvm_size 27917287424
elastic-01 10.0.5.22 10.0.5.22 zone zone1
elastic-01 10.0.5.22 10.0.5.22 transform.node true
elastic-02 10.0.5.23 10.0.5.23 ml.machine_memory 42407346176
elastic-02 10.0.5.23 10.0.5.23 ml.max_open_jobs 512
elastic-02 10.0.5.23 10.0.5.23 xpack.installed true
elastic-02 10.0.5.23 10.0.5.23 ml.max_jvm_size 27917287424
elastic-02 10.0.5.23 10.0.5.23 zone zone1
elastic-02 10.0.5.23 10.0.5.23 transform.node true
elastic-06 10.0.5.27 10.0.5.27 ml.machine_memory 42002579456
elastic-06 10.0.5.27 10.0.5.27 ml.max_open_jobs 512
elastic-06 10.0.5.27 10.0.5.27 xpack.installed true
elastic-06 10.0.5.27 10.0.5.27 ml.max_jvm_size 27917287424
elastic-06 10.0.5.27 10.0.5.27 zone zone2
elastic-06 10.0.5.217 10.0.5.27 transform.node true
elastic-11 10.0.25.24 10.0.25.24 ml.machine_memory 12428124160
elastic-11 10.0.25.24 10.0.25.24 ml.max_open_jobs 512
elastic-11 10.0.25.24 10.0.25.24 xpack.installed true
elastic-11 10.0.25.24 10.0.25.24 ml.max_jvm_size 2147483648
elastic-11 10.0.25.24 10.0.25.24 zone zone3
elastic-11 10.0.25.24 10.0.25.24 transform.node false
here is output from GET ilm/policy
{".alerts-ilm-policy":{"version":19,"modified_date":"2022-08-23T04:29:09.081Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},".deprecation-indexing-ilm-policy":{"version":1,"modified_date":"2022-01-12T18:09:22.265Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"10gb","max_age":"14d"}}}}},"in_use_by":{"indices":[".ds-.logs-deprecation.elasticsearch-default-2022.04.06-000007",".ds-.logs-deprecation.elasticsearch-default-2022.03.09-000005",".ds-.logs-deprecation.elasticsearch-default-2022.02.09-000003",".ds-.logs-deprecation.elasticsearch-default-2022.02.23-000004",".ds-.logs-deprecation.elasticsearch-default-2022.06.29-000013",".ds-.logs-deprecation.elasticsearch-default-2022.07.13-000014",".ds-.logs-deprecation.elasticsearch-default-2022.08.10-000016",".ds-.logs-deprecation.elasticsearch-default-2022.07.27-000015",".ds-.logs-deprecation.elasticsearch-default-2022.01.26-000002",".ds-.logs-deprecation.elasticsearch-default-2022.08.24-000017",".ds-.logs-deprecation.elasticsearch-default-2022.01.12-000001",".ds-.logs-deprecation.elasticsearch-default-2022.06.15-000012",".ds-.logs-deprecation.elasticsearch-default-2022.04.20-000008",".ds-.logs-deprecation.elasticsearch-default-2022.05.04-000009",".ds-.logs-deprecation.elasticsearch-default-2022.03.23-000006",".ds-.logs-deprecation.elasticsearch-default-2022.06.01-000011",".ds-.logs-deprecation.elasticsearch-default-2022.05.18-000010"],"data_streams":[".logs-deprecation.elasticsearch-default"],"composable_templates":[".deprecation-indexing-template"]}},".fleet-actions-results-ilm-policy":{"version":1,"modified_date":"2022-01-12T17:12:45.443Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"300gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},".items-forteforex":{"version":1,"modified_date":"2022-07-22T05:20:41.438Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[".items–000001"],"data_streams":[],"composable_templates":[]}},".lists":{"version":1,"modified_date":"2022-07-22T05:20:41.356Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[".lists-forteforex-000001"],"data_streams":[],"composable_templates":[]}},"180-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.121Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"180d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 180 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"30-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.229Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"30d","actions":{"delete":{"delete_searchable_snapshot":true}}},"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}}},"_meta":{"description":"built-in ILM policy using the hot and warm phases with a retention of 30 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"365-days-default":{"version":1,"modified_date":"2022-01-12T18:09:21.893Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"365d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 365 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"7-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.046Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"7d"}}},"delete":{"min_age":"7d","actions":{"delete":{"delete_searchable_snapshot":true}}},"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}}},"_meta":{"description":"built-in ILM policy using the hot and warm phases with a retention of 7 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"90-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.192Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 90 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"ilm-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.071Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":["ilm-history-2-000017","ilm-history-2-000016","ilm-history-2-000015","ilm-history-2-000014",".ds-ilm-history-5-2022.07.11-000010",".ds-ilm-history-5-2022.06.11-000008",".ds-ilm-history-5-2022.05.12-000006",".ds-ilm-history-5-2022.08.10-000012"],"data_streams":["ilm-history-5"],"composable_templates":["ilm-history"]}},"kibana-event-log-policy":{"version":1,"modified_date":"2021-04-23T06:56:32.286Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[".kibana-event-log-7.16.2-000005",".kibana-event-log-7.16.2-000007",".kibana-event-log-7.16.2-000006",".kibana-event-log-7.16.2-000008"],"data_streams":[],"composable_templates":[".kibana-event-log-7.16.2-template"]}},"kibana-reporting":{"version":1,"modified_date":"2022-01-12T18:12:47.764Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{}}}},"in_use_by":{"indices":[".reporting-2022-08-07",".reporting-2022-06-05",".reporting-2022-08-28",".reporting-2022-05-29",".reporting-2022-03-27",".reporting-2022-05-08",".reporting-2022-01-16",".reporting-2022-08-21",".reporting-2022-01-23",".reporting-2022-09-04",".reporting-2022-05-22"],"data_streams":[],"composable_templates":[]}},"logs":{"version":1,"modified_date":"2021-04-16T05:52:26.933Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["logs"]}},"metrics":{"version":1,"modified_date":"2021-04-16T05:52:26.979Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["metrics"]}},"ml-size-based-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:26.877Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[".ml-state",".ml-stats"]}},"slm-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.112Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[".slm-history-2-000014",".slm-history-2-000015",".slm-history-2-000016",".slm-history-2-000017"],"data_streams":[],"composable_templates":[".slm-history"]}},"synthetics":{"version":1,"modified_date":"2022-01-12T18:09:21.976Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}}},"_meta":{"description":"default policy for the synthetics index template installed by x-pack","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["synthetics"]}},"watch-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.024Z","policy":{"phases":{"delete":{"min_age":"7d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[".watch-history-13"]}}}
nothing criminal, right?
and yeah, i have 50 indices with docs.deleted and at the moment i do force_merge 1 by one of this indices
for example this is shards that created in 03.09.2022
cat teks
lile-abz-2022.09.03 0 r STARTED 1818053 565.8mb 10.0.5.28 elastic-07
lile-abz-2022.09.03 0 p STARTED 1818053 545.6mb 10.0.5.22 elastic-01
pari-kurma-market-a-2022.09.03 0 r STARTED 9258217 5.7gb 10.0.5.27 elastic-06
pari-kurma-market-a-2022.09.03 0 p STARTED 9258217 5.7gb 10.0.5.22 elastic-01
gavn-pyati-2022.09.03 0 r STARTED 33761356 14.7gb 10.0.5.27 elastic-06
gavn-pyati-2022.09.03 0 p STARTED 33761356 14.8gb 10.0.5.22 elastic-01
gavn-pyati-3-2022.09.03 0 r STARTED 9768830 3.9gb 10.0.5.28 elastic-07
gavn-pyati-3-2022.09.03 0 p STARTED 9768830 3.9gb 10.0.5.23 elastic-02
lola-chik-a-2022.09.03 0 r STARTED 9581756 3.2gb 10.0.5.27 elastic-06
lola-chik-a-2022.09.03 0 p STARTED 9581756 3.2gb 10.0.5.22 elastic-01
lila-xyu-a-2022.09.03 0 r STARTED 1441592 605.5mb 10.0.5.27 elastic-06
lila-xyu-a-2022.09.03 0 p STARTED 1441592 603.3mb 10.0.5.23 elastic-02
gavn-pyati-1-2022.09.03 0 r STARTED 10179423 3.9gb 10.0.5.27 elastic-06
gavn-pyati-1-2022.09.03 0 p STARTED 10179423 3.9gb 10.0.5.22 elastic-01
gorilla-energ-a-2022.09.03 0 r STARTED 19341369 26.6gb 10.0.5.28 elastic-07
gorilla-energ-a-2022.09.03 0 p STARTED 19341369 26.4gb 10.0.5.23 elastic-02
chih-bimochk-a-2022.09.03 1 r STARTED 23188624 22.5gb 10.0.5.27 elastic-06
chih-bimochk-a-2022.09.03 1 p STARTED 23188624 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 2 r STARTED 23181295 22.5gb 10.0.5.28 elastic-07
chih-bimochk-a-2022.09.03 2 p STARTED 23181295 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 3 r STARTED 23175665 22.5gb 10.0.5.27 elastic-06
chih-bimochk-a-2022.09.03 3 p STARTED 23175665 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 0 r STARTED 23182594 22.5gb 10.0.5.28 elastic-07
chih-bimochk-a-2022.09.03 0 p STARTED 23182594 22.4gb 10.0.5.22 elastic-01
prod-ca-a-2022.09.03 0 r STARTED 97339290 16.8gb 10.0.5.27 elastic-06
prod-ca-a-2022.09.03 0 p STARTED 97339290 16.8gb 10.0.5.22 elastic-01
jerusal-abracatd-a-2022.09.03 0 r STARTED 3647629 7.6gb 10.0.5.28 elastic-07
jerusal-abracatd-a-2022.09.03 0 p STARTED 3647629 7.6gb 10.0.5.22 elastic-01
prod-anstasia-a-2022.09.03 0 r STARTED 42794060 14.9gb 10.0.5.28 elastic-07
prod-anstasia-a-2022.09.03 0 p STARTED 42794060 14.9gb 10.0.5.22 elastic-01
sasha-log-2022.09.03 0 r STARTED 67602 6.5mb 10.0.5.28 elastic-07
sasha-log-2022.09.03 0 p STARTED 67602 6.5mb 10.0.5.23 elastic-02
prod-adedimitrious-a-2022.09.03 0 r STARTED 10874246 22.4gb 10.0.5.28 elastic-07
prod-adedimitrious-a-2022.09.03 0 p STARTED 10874246 22.4gb 10.0.5.22 elastic-01
as you can see almost all primaries located in elatic-01 with highest disk utiization, but nor on elastic-02 or other nodes
ADDED
i'm just create a index with 5 primary and 0 replicas and all of the primaries was allocated on elastic-01
curl -s -X GET http://10.0.5.22:9200/_cat/shards?v | grep swiss
i_want_swiss_passport 4 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 1 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 2 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 3 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 0 p STARTED 0 226b 10.0.5.22 elastic-01