elasticsearch in yellow status - elasticsearch

I have a two nodes elasticsearch cluster with logstash and kibana. The cluster was on green status until I have started the logstash the elasticsearch cluster went to yellow status.
running curl -XGET http://localhost:9200/_cat/shards?pretty
will show the following
.kibana 0 p STARTED 2 8.2kb 172.17.0.1 Ereshkigal
.kibana 0 r UNASSIGNED
logstash-2015.10.18 4 r STARTED 69 101.9kb 172.17.0.1 Ereshkigal
logstash-2015.10.18 4 p STARTED 69 101.7kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.18 0 r STARTED 62 65.7kb 172.17.0.1 Ereshkigal
logstash-2015.10.18 0 p STARTED 62 89.4kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.18 3 p STARTED 76 48.1kb 172.17.0.1 Ereshkigal
logstash-2015.10.18 3 r UNASSIGNED
logstash-2015.10.18 1 p STARTED 74 78.8kb 172.17.0.1 Ereshkigal
logstash-2015.10.18 1 r UNASSIGNED
logstash-2015.10.18 2 r STARTED 79 56.8kb 172.17.0.1 Ereshkigal
logstash-2015.10.18 2 p STARTED 79 65.1kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.19 4 p STARTED 7 43.4kb 172.17.0.1 Ereshkigal
logstash-2015.10.19 4 r UNASSIGNED
logstash-2015.10.19 0 r STARTED 7 50.8kb 172.17.0.1 Ereshkigal
logstash-2015.10.19 0 p STARTED 7 58.3kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.19 3 r STARTED 9 67.4kb 172.17.0.1 Ereshkigal
logstash-2015.10.19 3 p STARTED 9 67.3kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.19 1 r STARTED 12 76.4kb 172.17.0.1 Ereshkigal
logstash-2015.10.19 1 p STARTED 12 13.8kb 172.17.0.2 Hargen the Measurer
logstash-2015.10.19 2 p STARTED 13 78kb 172.17.0.1 Ereshkigal
logstash-2015.10.19 2 r UNASSIGNED
How can I get the cluster back in green status?


Yellow means you have unassigned shards, as shown by your output.
Make sure your shard allocation is enabled:
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
You can also try forcing the shard onto a node:
https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

Related

Elasticsearch allocates primary shards only on first node in the cluster

I have 5 node elasticsearch cluster to store logs ( 4 master eligble and data and 1 master only node)
and all primary shards that was created allocate only on firt node in the cluster so the 1st node almost totally overloaded by cpu
why this happens? nodes have the same specs (cpu,ram,etc)
here is my cat/allocation ( i manually move shards from elastic-01 every morning, and on elastic-01 is fewer shards that on the others, but anyway load on elastic-01 is big)
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
354 4.9tb 4.9tb 804.7gb 5.7tb 86 10.0.5.22 10.0.5.252 elastic-01
435 4tb 4tb 1.7tb 5.7tb 70 10.0.5.23 10.0.5.23 elastic-02
434 4.6tb 4.6tb 1tb 5.7tb 80 10.0.5.27 10.0.5.27 elastic-06
434 4.7tb 4.7tb 1014.8gb 5.7tb 82 10.0.5.28 10.0.5.28 elastic-07
and here is nodeattrs
node host ip attr value
elastic-07 10.0.5.28 10.0.5.28 ml.machine_memory 42002579456
elastic-07 10.0.5.28 10.0.5.28 ml.max_open_jobs 512
elastic-07 10.0.5.28 10.0.5.28 xpack.installed true
elastic-07 10.0.5.28 10.0.5.28 ml.max_jvm_size 27917287424
elastic-07 10.0.5.28 10.0.5.28 zone zone2
elastic-07 10.0.5.28 10.0.5.28 transform.node true
elastic-01 10.0.5.22 10.0.5.22 ml.machine_memory 42002583552
elastic-01 10.0.5.22 10.0.5.22 ml.max_open_jobs 512
elastic-01 10.0.5.22 10.0.5.22 xpack.installed true
elastic-01 10.0.5.22 10.0.5.22 ml.max_jvm_size 27917287424
elastic-01 10.0.5.22 10.0.5.22 zone zone1
elastic-01 10.0.5.22 10.0.5.22 transform.node true
elastic-02 10.0.5.23 10.0.5.23 ml.machine_memory 42407346176
elastic-02 10.0.5.23 10.0.5.23 ml.max_open_jobs 512
elastic-02 10.0.5.23 10.0.5.23 xpack.installed true
elastic-02 10.0.5.23 10.0.5.23 ml.max_jvm_size 27917287424
elastic-02 10.0.5.23 10.0.5.23 zone zone1
elastic-02 10.0.5.23 10.0.5.23 transform.node true
elastic-06 10.0.5.27 10.0.5.27 ml.machine_memory 42002579456
elastic-06 10.0.5.27 10.0.5.27 ml.max_open_jobs 512
elastic-06 10.0.5.27 10.0.5.27 xpack.installed true
elastic-06 10.0.5.27 10.0.5.27 ml.max_jvm_size 27917287424
elastic-06 10.0.5.27 10.0.5.27 zone zone2
elastic-06 10.0.5.217 10.0.5.27 transform.node true
elastic-11 10.0.25.24 10.0.25.24 ml.machine_memory 12428124160
elastic-11 10.0.25.24 10.0.25.24 ml.max_open_jobs 512
elastic-11 10.0.25.24 10.0.25.24 xpack.installed true
elastic-11 10.0.25.24 10.0.25.24 ml.max_jvm_size 2147483648
elastic-11 10.0.25.24 10.0.25.24 zone zone3
elastic-11 10.0.25.24 10.0.25.24 transform.node false
here is output from GET ilm/policy
{".alerts-ilm-policy":{"version":19,"modified_date":"2022-08-23T04:29:09.081Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},".deprecation-indexing-ilm-policy":{"version":1,"modified_date":"2022-01-12T18:09:22.265Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"10gb","max_age":"14d"}}}}},"in_use_by":{"indices":[".ds-.logs-deprecation.elasticsearch-default-2022.04.06-000007",".ds-.logs-deprecation.elasticsearch-default-2022.03.09-000005",".ds-.logs-deprecation.elasticsearch-default-2022.02.09-000003",".ds-.logs-deprecation.elasticsearch-default-2022.02.23-000004",".ds-.logs-deprecation.elasticsearch-default-2022.06.29-000013",".ds-.logs-deprecation.elasticsearch-default-2022.07.13-000014",".ds-.logs-deprecation.elasticsearch-default-2022.08.10-000016",".ds-.logs-deprecation.elasticsearch-default-2022.07.27-000015",".ds-.logs-deprecation.elasticsearch-default-2022.01.26-000002",".ds-.logs-deprecation.elasticsearch-default-2022.08.24-000017",".ds-.logs-deprecation.elasticsearch-default-2022.01.12-000001",".ds-.logs-deprecation.elasticsearch-default-2022.06.15-000012",".ds-.logs-deprecation.elasticsearch-default-2022.04.20-000008",".ds-.logs-deprecation.elasticsearch-default-2022.05.04-000009",".ds-.logs-deprecation.elasticsearch-default-2022.03.23-000006",".ds-.logs-deprecation.elasticsearch-default-2022.06.01-000011",".ds-.logs-deprecation.elasticsearch-default-2022.05.18-000010"],"data_streams":[".logs-deprecation.elasticsearch-default"],"composable_templates":[".deprecation-indexing-template"]}},".fleet-actions-results-ilm-policy":{"version":1,"modified_date":"2022-01-12T17:12:45.443Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"300gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},".items-forteforex":{"version":1,"modified_date":"2022-07-22T05:20:41.438Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[".items–000001"],"data_streams":[],"composable_templates":[]}},".lists":{"version":1,"modified_date":"2022-07-22T05:20:41.356Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[".lists-forteforex-000001"],"data_streams":[],"composable_templates":[]}},"180-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.121Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"180d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 180 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"30-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.229Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"30d","actions":{"delete":{"delete_searchable_snapshot":true}}},"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}}},"_meta":{"description":"built-in ILM policy using the hot and warm phases with a retention of 30 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"365-days-default":{"version":1,"modified_date":"2022-01-12T18:09:21.893Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"365d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 365 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"7-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.046Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"7d"}}},"delete":{"min_age":"7d","actions":{"delete":{"delete_searchable_snapshot":true}}},"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}}},"_meta":{"description":"built-in ILM policy using the hot and warm phases with a retention of 7 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"90-days-default":{"version":1,"modified_date":"2022-01-12T18:09:22.192Z","policy":{"phases":{"warm":{"min_age":"2d","actions":{"forcemerge":{"max_num_segments":1},"shrink":{"number_of_shards":1}}},"cold":{"min_age":"30d","actions":{}},"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}},"_meta":{"description":"built-in ILM policy using the hot, warm, and cold phases with a retention of 90 days","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[]}},"ilm-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.071Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":["ilm-history-2-000017","ilm-history-2-000016","ilm-history-2-000015","ilm-history-2-000014",".ds-ilm-history-5-2022.07.11-000010",".ds-ilm-history-5-2022.06.11-000008",".ds-ilm-history-5-2022.05.12-000006",".ds-ilm-history-5-2022.08.10-000012"],"data_streams":["ilm-history-5"],"composable_templates":["ilm-history"]}},"kibana-event-log-policy":{"version":1,"modified_date":"2021-04-23T06:56:32.286Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[".kibana-event-log-7.16.2-000005",".kibana-event-log-7.16.2-000007",".kibana-event-log-7.16.2-000006",".kibana-event-log-7.16.2-000008"],"data_streams":[],"composable_templates":[".kibana-event-log-7.16.2-template"]}},"kibana-reporting":{"version":1,"modified_date":"2022-01-12T18:12:47.764Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{}}}},"in_use_by":{"indices":[".reporting-2022-08-07",".reporting-2022-06-05",".reporting-2022-08-28",".reporting-2022-05-29",".reporting-2022-03-27",".reporting-2022-05-08",".reporting-2022-01-16",".reporting-2022-08-21",".reporting-2022-01-23",".reporting-2022-09-04",".reporting-2022-05-22"],"data_streams":[],"composable_templates":[]}},"logs":{"version":1,"modified_date":"2021-04-16T05:52:26.933Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["logs"]}},"metrics":{"version":1,"modified_date":"2021-04-16T05:52:26.979Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["metrics"]}},"ml-size-based-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:26.877Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb"}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[".ml-state",".ml-stats"]}},"slm-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.112Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_size":"50gb","max_age":"30d"}}},"delete":{"min_age":"90d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[".slm-history-2-000014",".slm-history-2-000015",".slm-history-2-000016",".slm-history-2-000017"],"data_streams":[],"composable_templates":[".slm-history"]}},"synthetics":{"version":1,"modified_date":"2022-01-12T18:09:21.976Z","policy":{"phases":{"hot":{"min_age":"0ms","actions":{"rollover":{"max_primary_shard_size":"50gb","max_age":"30d"}}}},"_meta":{"description":"default policy for the synthetics index template installed by x-pack","managed":true}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":["synthetics"]}},"watch-history-ilm-policy":{"version":1,"modified_date":"2021-04-16T05:52:27.024Z","policy":{"phases":{"delete":{"min_age":"7d","actions":{"delete":{"delete_searchable_snapshot":true}}}}},"in_use_by":{"indices":[],"data_streams":[],"composable_templates":[".watch-history-13"]}}}
nothing criminal, right?
and yeah, i have 50 indices with docs.deleted and at the moment i do force_merge 1 by one of this indices
for example this is shards that created in 03.09.2022
cat teks
lile-abz-2022.09.03 0 r STARTED 1818053 565.8mb 10.0.5.28 elastic-07
lile-abz-2022.09.03 0 p STARTED 1818053 545.6mb 10.0.5.22 elastic-01
pari-kurma-market-a-2022.09.03 0 r STARTED 9258217 5.7gb 10.0.5.27 elastic-06
pari-kurma-market-a-2022.09.03 0 p STARTED 9258217 5.7gb 10.0.5.22 elastic-01
gavn-pyati-2022.09.03 0 r STARTED 33761356 14.7gb 10.0.5.27 elastic-06
gavn-pyati-2022.09.03 0 p STARTED 33761356 14.8gb 10.0.5.22 elastic-01
gavn-pyati-3-2022.09.03 0 r STARTED 9768830 3.9gb 10.0.5.28 elastic-07
gavn-pyati-3-2022.09.03 0 p STARTED 9768830 3.9gb 10.0.5.23 elastic-02
lola-chik-a-2022.09.03 0 r STARTED 9581756 3.2gb 10.0.5.27 elastic-06
lola-chik-a-2022.09.03 0 p STARTED 9581756 3.2gb 10.0.5.22 elastic-01
lila-xyu-a-2022.09.03 0 r STARTED 1441592 605.5mb 10.0.5.27 elastic-06
lila-xyu-a-2022.09.03 0 p STARTED 1441592 603.3mb 10.0.5.23 elastic-02
gavn-pyati-1-2022.09.03 0 r STARTED 10179423 3.9gb 10.0.5.27 elastic-06
gavn-pyati-1-2022.09.03 0 p STARTED 10179423 3.9gb 10.0.5.22 elastic-01
gorilla-energ-a-2022.09.03 0 r STARTED 19341369 26.6gb 10.0.5.28 elastic-07
gorilla-energ-a-2022.09.03 0 p STARTED 19341369 26.4gb 10.0.5.23 elastic-02
chih-bimochk-a-2022.09.03 1 r STARTED 23188624 22.5gb 10.0.5.27 elastic-06
chih-bimochk-a-2022.09.03 1 p STARTED 23188624 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 2 r STARTED 23181295 22.5gb 10.0.5.28 elastic-07
chih-bimochk-a-2022.09.03 2 p STARTED 23181295 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 3 r STARTED 23175665 22.5gb 10.0.5.27 elastic-06
chih-bimochk-a-2022.09.03 3 p STARTED 23175665 22.4gb 10.0.5.22 elastic-01
chih-bimochk-a-2022.09.03 0 r STARTED 23182594 22.5gb 10.0.5.28 elastic-07
chih-bimochk-a-2022.09.03 0 p STARTED 23182594 22.4gb 10.0.5.22 elastic-01
prod-ca-a-2022.09.03 0 r STARTED 97339290 16.8gb 10.0.5.27 elastic-06
prod-ca-a-2022.09.03 0 p STARTED 97339290 16.8gb 10.0.5.22 elastic-01
jerusal-abracatd-a-2022.09.03 0 r STARTED 3647629 7.6gb 10.0.5.28 elastic-07
jerusal-abracatd-a-2022.09.03 0 p STARTED 3647629 7.6gb 10.0.5.22 elastic-01
prod-anstasia-a-2022.09.03 0 r STARTED 42794060 14.9gb 10.0.5.28 elastic-07
prod-anstasia-a-2022.09.03 0 p STARTED 42794060 14.9gb 10.0.5.22 elastic-01
sasha-log-2022.09.03 0 r STARTED 67602 6.5mb 10.0.5.28 elastic-07
sasha-log-2022.09.03 0 p STARTED 67602 6.5mb 10.0.5.23 elastic-02
prod-adedimitrious-a-2022.09.03 0 r STARTED 10874246 22.4gb 10.0.5.28 elastic-07
prod-adedimitrious-a-2022.09.03 0 p STARTED 10874246 22.4gb 10.0.5.22 elastic-01
as you can see almost all primaries located in elatic-01 with highest disk utiization, but nor on elastic-02 or other nodes
ADDED
i'm just create a index with 5 primary and 0 replicas and all of the primaries was allocated on elastic-01
curl -s -X GET http://10.0.5.22:9200/_cat/shards?v | grep swiss
i_want_swiss_passport 4 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 1 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 2 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 3 p STARTED 0 226b 10.0.5.22 elastic-01
i_want_swiss_passport 0 p STARTED 0 226b 10.0.5.22 elastic-01

Elasticsearch one specific shard keep initializing in different data nodes

I am getting ElasticsearchStatusWarning saying that the cluster state is yellow. Upon running the health check API, I see below
curl -X GET http://localhost:9200/_cluster/health/
{"cluster_name":"my-elasticsearch","status":"yellow","timed_out":false,"number_of_nodes":8,"number_of_data_nodes":3,"active_primary_shards":220,"active_shards":438,"relocating_shards":0,"initializing_shards":2,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":99.54545454545455}
initializing_shards is 2. So, I further run the below call
curl -X GET
http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
|grep INIT
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 33457 100 33457 0 graph_vertex_24_18549 0 r INITIALIZING ALLOCATION_FAILED
0 79609 0 --:--:-- --:--:-- --:--:-- 79659
curl -X GET http://localhost:9200/_cat/shards/graph_vertex_24_18549
graph_vertex_24_18549 0 p STARTED 8373375 8.4gb IP1 elasticsearch-data-1
graph_vertex_24_18549 0 r INITIALIZING IP2 elasticsearch-data-2
And rerunning the same command in few mins, shows now it's being initialized in elasticsearch-data-0. See below
graph_vertex_24_18549 0 p STARTED 8373375 8.4gb IP1 elasticsearch-data-1
graph_vertex_24_18549 0 r INITIALIZING IP0 elasticsearch-data-0
If i rerun it again in few mins, I can see it's again being initialized in elasticsearch-data-2 again. But it never gets STARTED.
curl -X GET http://localhost:9200/_cat/allocation?v
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
147 162.2gb 183.8gb 308.1gb 492gb 37 IP1 IP1 elasticsearch-data-2
146 217.3gb 234.2gb 257.7gb 492gb 47 IP2 IP2 elasticsearch-data-1
147 216.6gb 231.2gb 260.7gb 492gb 47 IP3 IP3 elasticsearch-data-0
curl -X GET http://localhost:9200/_cat/nodes?v
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
IP1 7 77 20 4.17 4.57 4.88 mi - elasticsearch-master-2
IP2 72 59 7 2.59 2.38 2.19 i - elasticsearch-5f4bd5b88f-4lvxz
IP3 57 49 3 0.75 1.13 1.09 di - elasticsearch-data-2
IP4 63 57 21 2.69 3.58 4.11 di - elasticsearch-data-0
IP5 5 59 7 2.59 2.38 2.19 mi - elasticsearch-master-0
IP6 69 53 13 4.67 4.60 4.66 di - elasticsearch-data-1
IP7 8 70 14 2.86 3.20 3.09 mi * elasticsearch-master-1
IP8 30 77 20 4.17 4.57 4.88 i - elasticsearch-5f4bd5b88f-wnrl4
curl -s -XGET http://localhost:9200/_cluster/allocation/explain -d '{
"index": "graph_vertex_24_18549", "shard":
0, "primary": false }' -H 'Content-type: application/json'
{"index":"graph_vertex_24_18549","shard":0,"primary":false,"current_state":"initializing","unassigned_info":{"reason":"ALLOCATION_FAILED","at":"2020-11-04T08:21:45.756Z","failed_allocation_attempts":1,"details":"failed shard on node [1XEXS92jTK-wwanNgQrxsA]: failed to perform indices:data/write/bulk[s] on replica [graph_vertex_24_18549][0], node[1XEXS92jTK-wwanNgQrxsA], [R], s[STARTED], a[id=RnTOlfQuQkOumVuw_NeuTw], failure RemoteTransportException[[elasticsearch-data-2][IP:9300][indices:data/write/bulk[s][r]]]; nested: CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [4322682690/4gb], which is larger than the limit of [4005632409/3.7gb], real usage: [3646987112/3.3gb], new bytes reserved: [675695578/644.3mb]]; ","last_allocation_status":"no_attempt"},"current_node":{"id":"o_9jyrmOSca9T12J4bY0Nw","name":"elasticsearch-data-0","transport_address":"IP:9300"},"explanation":"the shard is in the process of initializing on node [elasticsearch-data-0], wait until initialization has completed"}
Thing is I was earlier getting alerted for Unassigned Shards due to the same exception as above - "CircuitBreakingException[[parent] Data too large, data for [<transport_request>] would be [4322682690/4gb], which is larger than the limit of [4005632409/3.7gb]"
But back then heap was only 2G. I increased it to 4G. And now I am seeing same error, but this time with respect to Initialising shards instead of Unallocated shards.
How can I remediate this?

Starting multiple nodes do not allocate replica shards

So, I had just started out with Elasticsearch on my local machine.
I have started 5 instances of Elasticsearch nodes. (simple ./bin/elasticsearch)
curl -s 'localhost:9200/_cat/nodes?v' gives:
host ip heap.percent ram.percent load node.role master name
127.0.0.1 127.0.0.1 5 99 3.13 d m Shirow Ishihara
127.0.0.1 127.0.0.1 7 100 3.13 d m Madame Web
127.0.0.1 127.0.0.1 5 100 3.13 d m Anthropomorpho
127.0.0.1 127.0.0.1 5 100 3.13 d m Paste-Pot Pete
127.0.0.1 127.0.0.1 2 100 3.13 d * Mephisto
My index has 2 primary shards and 5 replicas (total 10 replicas).
I had read that ES automatically scales horizontally and assigns/moves shards to new nodes. However, still all the 10 replicas are unassigned and both the 2 primary shards are in the same node.
curl -s 'localhost:9200/_cat/allocation?v' gives:
shards disk.indices disk.used disk.avail disk.total disk.percent host ip node
0 0b 105.5gb 6.2gb 111.8gb 94 127.0.0.1 127.0.0.1 Shirow Ishihara
0 0b 105.5gb 6.2gb 111.8gb 94 127.0.0.1 127.0.0.1 Paste-Pot Pete
2 318b 105.5gb 6.2gb 111.8gb 94 127.0.0.1 127.0.0.1 Mephisto
0 0b 105.5gb 6.2gb 111.8gb 94 127.0.0.1 127.0.0.1 Anthropomorpho
0 0b 105.5gb 6.2gb 111.8gb 94 127.0.0.1 127.0.0.1 Madame Web
10 UNASSIGNED

You have too few available disk space an ES is actually trying to move away some shards. But all your nodes are on the same machine so there is nowhere else where to move them and they stay unassigned. The used disk space is more than 90% of the total disk size and ES is hitting the the high watermark.
Read here more about this.

Elasticsearch is using way too much disk space per document

I'am using a 1.7 Elastic Search and try to index documents in it.
With 23 000 documents my index size is 17Go... This seems a way too large for only 23K docs (as my doc, in json, are around 13ko).
My doc have a lot of compound doc in it (the 13Ko is for all, doc + compound).
I keep the _all in my doc (and I need it).
I use this nGram tokeniser (maybe 2 is too low?)
'min_gram' => 2,
'max_gram' => 20
How a 13Ko per document become a 775Ko per doc after being added to Elastic?
Here a sample of my shards, where I had indexed only 1K doc :
dyb-fr_fr 4 p STARTED 7188 263.1mb 10.20.40.29 Doyoubuzz
dyb-fr_fr 0 p STARTED 7675 258.6mb 10.20.40.29 Doyoubuzz
dyb-fr_fr 3 p STARTED 7268 258.5mb 10.20.40.29 Doyoubuzz
dyb-fr_fr 1 p STARTED 8560 300.1mb 10.20.40.29 Doyoubuzz
dyb-fr_fr 2 p STARTED 7287 244.3mb 10.20.40.29 Doyoubuzz
And by going deeper on segments :
index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound
dyb-fr_fr 0 p 127.0.0.1 _2 2 291 0 9.7mb 173746 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _7 7 57 0 2.4mb 52650 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _d 13 43 0 2.8mb 71242 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _e 14 322 0 11.2mb 197706 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _f 15 1912 0 64.8mb 928522 false true 4.10.4 false
dyb-fr_fr 0 p 127.0.0.1 _k 20 64 0 1.4mb 43090 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _p 25 12 0 170.5kb 17794 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _t 29 1612 0 48.5mb 692322 false true 4.10.4 false
dyb-fr_fr 0 p 127.0.0.1 _y 34 228 0 6.9mb 128042 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _z 35 159 0 6.3mb 121266 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _13 39 232 0 6.4mb 125386 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _15 41 127 0 4.2mb 97738 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _16 42 1675 0 44.3mb 637266 false true 4.10.4 false
dyb-fr_fr 0 p 127.0.0.1 _17 43 203 0 4.3mb 92282 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _18 44 146 0 5.4mb 108730 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _19 45 236 0 6.3mb 115474 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _1a 46 63 0 2.1mb 52762 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _1b 47 118 0 4.2mb 88050 false true 4.10.4 true
dyb-fr_fr 0 p 127.0.0.1 _1c 48 175 0 5.4mb 105570 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _2 2 89 0 2mb 56578 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _7 7 49 0 1.9mb 49810 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _9 9 314 0 10.6mb 184426 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _b 11 139 0 2.7mb 66218 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _f 15 1634 0 63.3mb 916226 false true 4.10.4 false
dyb-fr_fr 1 p 127.0.0.1 _g 16 72 0 1.3mb 48850 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _n 23 67 0 2.1mb 56826 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _o 24 43 0 1009.2kb 32458 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _u 30 2097 0 55.2mb 770266 false true 4.10.4 false
dyb-fr_fr 1 p 127.0.0.1 _x 33 35 0 877.5kb 29978 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _12 38 114 0 1.9mb 46818 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _15 41 292 0 5.7mb 116850 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _16 42 2264 0 64.6mb 923826 false true 4.10.4 false
dyb-fr_fr 1 p 127.0.0.1 _17 43 193 0 5.4mb 110674 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _18 44 79 0 1.8mb 44858 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _19 45 198 0 7mb 136298 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _1a 46 170 0 5.7mb 118922 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _1b 47 173 0 6.8mb 130610 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _1c 48 162 0 3.7mb 79610 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _1d 49 205 0 7.2mb 130818 false true 4.10.4 true
dyb-fr_fr 1 p 127.0.0.1 _1e 50 171 0 5.5mb 117946 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _0 0 404 0 16mb 270562 true true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _4 4 67 0 1.4mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _b 11 12 0 168.7kb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _f 15 24 0 245.3kb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _g 16 2003 0 57.6mb 808882 true true 4.10.4 false
dyb-fr_fr 2 p 127.0.0.1 _j 19 67 0 3.4mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _l 21 100 0 2mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _o 24 136 0 3.4mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _s 28 28 0 396.9kb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _t 29 2149 0 57.6mb 822498 true true 4.10.4 false
dyb-fr_fr 2 p 127.0.0.1 _u 30 171 0 5.7mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _v 31 144 0 7mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _w 32 123 0 2.7mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _x 33 147 0 6.2mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _y 34 129 0 4.3mb 0 true false 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _13 39 32 0 1.8mb 44994 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _14 40 614 0 14mb 233474 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1f 51 44 0 676.9kb 26306 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1g 52 42 0 1.3mb 35514 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1h 53 10 0 226.5kb 19578 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1j 55 30 0 1.1mb 43706 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1k 56 53 0 1.1mb 42610 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1o 60 234 0 5.7mb 113986 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1p 61 1366 0 43mb 626498 false true 4.10.4 false
dyb-fr_fr 2 p 127.0.0.1 _1q 62 5 0 118.6kb 13162 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1r 63 78 0 7.7mb 144554 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1s 64 213 0 5.4mb 112610 false true 4.10.4 true
dyb-fr_fr 2 p 127.0.0.1 _1t 65 10 0 314kb 18122 false true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _e 14 1873 0 63.9mb 915586 true true 4.10.4 false
dyb-fr_fr 3 p 127.0.0.1 _g 16 75 0 1.6mb 40210 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _k 20 20 0 242.4kb 20426 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _o 24 74 0 2.1mb 58242 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _p 25 13 0 324.8kb 22514 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _s 28 1770 0 44.1mb 636786 true true 4.10.4 false
dyb-fr_fr 3 p 127.0.0.1 _t 29 268 0 9.5mb 171306 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _u 30 26 0 837.3kb 27474 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _v 31 50 0 3mb 73322 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _10 36 78 0 2.6mb 69178 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _14 40 306 0 6.7mb 129666 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _15 41 1751 0 58.8mb 849858 true true 4.10.4 false
dyb-fr_fr 3 p 127.0.0.1 _16 42 138 0 6.3mb 134242 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _17 43 37 0 512.7kb 24842 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _18 44 58 0 1.7mb 52410 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _19 45 286 0 5.3mb 99234 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _1a 46 202 0 7.3mb 132914 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _1b 47 53 0 1.6mb 43634 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _1c 48 111 0 3.9mb 88362 true true 4.10.4 true
dyb-fr_fr 3 p 127.0.0.1 _1d 49 79 0 1.1mb 33410 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _4 4 66 0 1.6mb 41970 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _a 10 66 0 1mb 43786 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _c 12 269 0 17.4mb 296738 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _e 14 53 0 1.1mb 33954 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _f 15 2099 0 62.3mb 866978 true true 4.10.4 false
dyb-fr_fr 4 p 127.0.0.1 _k 20 52 0 1mb 42882 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _r 27 71 0 2.4mb 51186 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _s 28 2016 0 48.8mb 689106 true true 4.10.4 false
dyb-fr_fr 4 p 127.0.0.1 _x 33 36 0 1.6mb 48210 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _10 36 247 0 2.8mb 56842 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _12 38 258 0 10.9mb 189882 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _15 41 5 0 145.7kb 13386 true true 4.10.4 true
dyb-fr_fr 4 p 127.0.0.1 _16 42 1950 0 53.3mb 749626 true true 4.10.4 false

ngrams are producing a lot of tokens.
If you index "abcdef", you will index actually:
ab,bc,cd,de,ef,abc,bcd,cde,def,abcd,bcde,cdef,abcde,bcdef,abcdef
It's even worse if you have very big strings as the combinatory explodes.

rownum confused result

when i code
select
row_.*, rownum rownum_
from
(select
topic0_.id as id6_, topic0_.title as title6_,
topic0_.publish_author_id as publish3_6_, topic0_.reply_number as reply4_6_,
topic0_.read_number as read5_6_, topic0_.publish_author_name as publish6_6_,
topic0_.publish_time as publish7_6_, topic0_.last_reply_time as last8_6_,
topic0_.last_reply_name as last9_6_, topic0_.main_text as main10_6_, topic0_.FORUM_ID as FORUM11_6_
from topic topic0_
where topic0_.FORUM_ID='16F37E59FE494AC9B17A2340387335EF'
order by
case when topic0_.last_reply_time is not null then topic0_.last_reply_time
else topic0_.publish_time end desc
) row_;
the result:
ID6_ TITLE6_ PUBLISH3_6_ REPLY4_6_ READ5_6_ PUBLISH6_6_ PUBLISH7_6_ LAST8_6_ LAST9_6_ MAIN10_6_ FORUM11_6_ ROWNUM_
1 402840713e108ea4013e1090c95c0001 trytr 2 2 0 ss 2013-4-16 9:58:44 2013-4-16 9:59:21 ss fdsfds 16F37E59FE494AC9B17A2340387335EF 1
2 402840713e108ea4013e1090127e0000 trytr 2 0 0 ss 2013-4-16 9:57:57 ytrytr 16F37E59FE494AC9B17A2340387335EF 2
3 402840713e0d3616013e0d3618c20000 cde0 2 0 0 ss 2013-4-15 18:20:49 abc0 16F37E59FE494AC9B17A2340387335EF 3
4 402840713e0d3616013e0d3618da0063 cde99 2 0 0 ss 2013-4-15 18:20:49 abc99 16F37E59FE494AC9B17A2340387335EF 4
5 402840713e0d3616013e0d3618da0062 cde98 2 0 0 ss 2013-4-15 18:20:49 abc98 16F37E59FE494AC9B17A2340387335EF 5
6 402840713e0d3616013e0d3618da0061 cde97 2 0 0 ss 2013-4-15 18:20:49 abc97 16F37E59FE494AC9B17A2340387335EF 6
7 402840713e0d3616013e0d3618da0060 cde96 2 0 0 ss 2013-4-15 18:20:49 abc96 16F37E59FE494AC9B17A2340387335EF 7
8 402840713e0d3616013e0d3618da005f cde95 2 0 0 ss 2013-4-15 18:20:49 abc95 16F37E59FE494AC9B17A2340387335EF 8
9 402840713e0d3616013e0d3618da005e cde94 2 0 0 ss 2013-4-15 18:20:49 abc94 16F37E59FE494AC9B17A2340387335EF 9
10 402840713e0d3616013e0d3618da005d cde93 2 0 0 ss 2013-4-15 18:20:49 abc93 16F37E59FE494AC9B17A2340387335EF 10
11 402840713e0d3616013e0d3618da005c cde92 2 0 0 ss 2013-4-15 18:20:49 abc92 16F37E59FE494AC9B17A2340387335EF 11
12 402840713e0d3616013e0d3618da005b cde91 2 0 0 ss 2013-4-15 18:20:49 abc91 16F37E59FE494AC9B17A2340387335EF 12
13 402840713e0d3616013e0d3618d9005a cde90 2 0 0 ss 2013-4-15 18:20:49 abc90 16F37E59FE494AC9B17A2340387335EF 13
14 402840713e0d3616013e0d3618d90059 cde89 2 0 0 ss 2013-4-15 18:20:49 abc89 16F37E59FE494AC9B17A2340387335EF 14
15 402840713e0d3616013e0d3618d90058 cde88 2 0 0 ss 2013-4-15 18:20:49 abc88 16F37E59FE494AC9B17A2340387335EF 15
16 402840713e0d3616013e0d3618d90057 cde87 2 0 0 ss 2013-4-15 18:20:49 abc87 16F37E59FE494AC9B17A2340387335EF 16
17 402840713e0d3616013e0d3618d90056 cde86 2 0 0 ss 2013-4-15 18:20:49 abc86 16F37E59FE494AC9B17A2340387335EF 17
18 402840713e0d3616013e0d3618d90055 cde85 2 0 0 ss 2013-4-15 18:20:49 abc85 16F37E59FE494AC9B17A2340387335EF 18
19 402840713e0d3616013e0d3618d90054 cde84 2 0 0 ss 2013-4-15 18:20:49 abc84 16F37E59FE494AC9B17A2340387335EF 19
20 402840713e0d3616013e0d3618d90053 cde83 2 0 0 ss 2013-4-15 18:20:49 abc83 16F37E59FE494AC9B17A2340387335EF 20
21 402840713e0d3616013e0d3618d90052 cde82 2 0 0 ss 2013-4-15 18:20:49 abc82 16F37E59FE494AC9B17A2340387335EF 21
22 402840713e0d3616013e0d3618d90051 cde81 2 0 0 ss 2013-4-15 18:20:49 abc81 16F37E59FE494AC9B17A2340387335EF 22
23 402840713e0d3616013e0d3618d80050 cde80 2 0 0 ss 2013-4-15 18:20:49 abc80 16F37E59FE494AC9B17A2340387335EF 23
24 402840713e0d3616013e0d3618d8004f cde79 2 0 0 ss 2013-4-15 18:20:49 abc79 16F37E59FE494AC9B17A2340387335EF 24
25 402840713e0d3616013e0d3618d8004e cde78 2 0 0 ss 2013-4-15 18:20:49 abc78 16F37E59FE494AC9B17A2340387335EF 25
26 402840713e0d3616013e0d3618d8004d cde77 2 0 0 ss 2013-4-15 18:20:49 abc77 16F37E59FE494AC9B17A2340387335EF 26
27 402840713e0d3616013e0d3618d8004c cde76 2 0 0 ss 2013-4-15 18:20:49 abc76 16F37E59FE494AC9B17A2340387335EF 27
28 402840713e0d3616013e0d3618d8004b cde75 2 0 0 ss 2013-4-15 18:20:49 abc75 16F37E59FE494AC9B17A2340387335EF 28
29 402840713e0d3616013e0d3618d8004a cde74 2 0 0 ss 2013-4-15 18:20:49 abc74 16F37E59FE494AC9B17A2340387335EF 29
30 402840713e0d3616013e0d3618d80049 cde73 2 0 0 ss 2013-4-15 18:20:49 abc73 16F37E59FE494AC9B17A2340387335EF 30
31 402840713e0d3616013e0d3618d80048 cde72 2 0 0 ss 2013-4-15 18:20:49 abc72 16F37E59FE494AC9B17A2340387335EF 31
32 402840713e0d3616013e0d3618d70047 cde71 2 0 0 ss 2013-4-15 18:20:49 abc71 16F37E59FE494AC9B17A2340387335EF 32
33 402840713e0d3616013e0d3618d70046 cde70 2 0 0 ss 2013-4-15 18:20:49 abc70 16F37E59FE494AC9B17A2340387335EF 33
34 402840713e0d3616013e0d3618d70045 cde69 2 0 0 ss 2013-4-15 18:20:49 abc69 16F37E59FE494AC9B17A2340387335EF 34
35 402840713e0d3616013e0d3618d70044 cde68 2 0 0 ss 2013-4-15 18:20:49 abc68 16F37E59FE494AC9B17A2340387335EF 35
36 402840713e0d3616013e0d3618d70043 cde67 2 0 0 ss 2013-4-15 18:20:49 abc67 16F37E59FE494AC9B17A2340387335EF 36
37 402840713e0d3616013e0d3618d70042 cde66 2 0 0 ss 2013-4-15 18:20:49 abc66 16F37E59FE494AC9B17A2340387335EF 37
38 402840713e0d3616013e0d3618d70041 cde65 2 0 0 ss 2013-4-15 18:20:49 abc65 16F37E59FE494AC9B17A2340387335EF 38
39 402840713e0d3616013e0d3618d70040 cde64 2 0 0 ss 2013-4-15 18:20:49 abc64 16F37E59FE494AC9B17A2340387335EF 39
40 402840713e0d3616013e0d3618d7003f cde63 2 0 0 ss 2013-4-15 18:20:49 abc63 16F37E59FE494AC9B17A2340387335EF 40
41 402840713e0d3616013e0d3618d7003e cde62 2 0 0 ss 2013-4-15 18:20:49 abc62 16F37E59FE494AC9B17A2340387335EF 41
42 402840713e0d3616013e0d3618d6003d cde61 2 0 0 ss 2013-4-15 18:20:49 abc61 16F37E59FE494AC9B17A2340387335EF 42
43 402840713e0d3616013e0d3618d6003c cde60 2 0 0 ss 2013-4-15 18:20:49 abc60 16F37E59FE494AC9B17A2340387335EF 43
44 402840713e0d3616013e0d3618d6003b cde59 2 0 0 ss 2013-4-15 18:20:49 abc59 16F37E59FE494AC9B17A2340387335EF 44
45 402840713e0d3616013e0d3618d6003a cde58 2 0 0 ss 2013-4-15 18:20:49 abc58 16F37E59FE494AC9B17A2340387335EF 45
46 402840713e0d3616013e0d3618d60039 cde57 2 0 0 ss 2013-4-15 18:20:49 abc57 16F37E59FE494AC9B17A2340387335EF 46
47 402840713e0d3616013e0d3618d60038 cde56 2 0 0 ss 2013-4-15 18:20:49 abc56 16F37E59FE494AC9B17A2340387335EF 47
48 402840713e0d3616013e0d3618d60037 cde55 2 0 0 ss 2013-4-15 18:20:49 abc55 16F37E59FE494AC9B17A2340387335EF 48
49 402840713e0d3616013e0d3618d60036 cde54 2 0 0 ss 2013-4-15 18:20:49 abc54 16F37E59FE494AC9B17A2340387335EF 49
50 402840713e0d3616013e0d3618d60035 cde53 2 0 0 ss 2013-4-15 18:20:49 abc53 16F37E59FE494AC9B17A2340387335EF 50
51 402840713e0d3616013e0d3618d50034 cde52 2 0 0 ss 2013-4-15 18:20:49 abc52 16F37E59FE494AC9B17A2340387335EF 51
52 402840713e0d3616013e0d3618d50033 cde51 2 0 0 ss 2013-4-15 18:20:49 abc51 16F37E59FE494AC9B17A2340387335EF 52
53 402840713e0d3616013e0d3618d50032 cde50 2 0 0 ss 2013-4-15 18:20:49 abc50 16F37E59FE494AC9B17A2340387335EF 53
54 402840713e0d3616013e0d3618d50031 cde49 2 0 0 ss 2013-4-15 18:20:49 abc49 16F37E59FE494AC9B17A2340387335EF 54
55 402840713e0d3616013e0d3618d50030 cde48 2 0 0 ss 2013-4-15 18:20:49 abc48 16F37E59FE494AC9B17A2340387335EF 55
56 402840713e0d3616013e0d3618d5002f cde47 2 0 0 ss 2013-4-15 18:20:49 abc47 16F37E59FE494AC9B17A2340387335EF 56
57 402840713e0d3616013e0d3618d5002e cde46 2 0 0 ss 2013-4-15 18:20:49 abc46 16F37E59FE494AC9B17A2340387335EF 57
58 402840713e0d3616013e0d3618d5002d cde45 2 0 0 ss 2013-4-15 18:20:49 abc45 16F37E59FE494AC9B17A2340387335EF 58
59 402840713e0d3616013e0d3618d5002c cde44 2 0 0 ss 2013-4-15 18:20:49 abc44 16F37E59FE494AC9B17A2340387335EF 59
60 402840713e0d3616013e0d3618d5002b cde43 2 0 0 ss 2013-4-15 18:20:49 abc43 16F37E59FE494AC9B17A2340387335EF 60
61 402840713e0d3616013e0d3618d5002a cde42 2 0 0 ss 2013-4-15 18:20:49 abc42 16F37E59FE494AC9B17A2340387335EF 61
62 402840713e0d3616013e0d3618d40029 cde41 2 0 0 ss 2013-4-15 18:20:49 abc41 16F37E59FE494AC9B17A2340387335EF 62
63 402840713e0d3616013e0d3618d40028 cde40 2 0 0 ss 2013-4-15 18:20:49 abc40 16F37E59FE494AC9B17A2340387335EF 63
64 402840713e0d3616013e0d3618d40027 cde39 2 0 0 ss 2013-4-15 18:20:49 abc39 16F37E59FE494AC9B17A2340387335EF 64
65 402840713e0d3616013e0d3618d40026 cde38 2 0 0 ss 2013-4-15 18:20:49 abc38 16F37E59FE494AC9B17A2340387335EF 65
66 402840713e0d3616013e0d3618d40025 cde37 2 0 0 ss 2013-4-15 18:20:49 abc37 16F37E59FE494AC9B17A2340387335EF 66
67 402840713e0d3616013e0d3618d40024 cde36 2 0 0 ss 2013-4-15 18:20:49 abc36 16F37E59FE494AC9B17A2340387335EF 67
68 402840713e0d3616013e0d3618d40023 cde35 2 0 0 ss 2013-4-15 18:20:49 abc35 16F37E59FE494AC9B17A2340387335EF 68
69 402840713e0d3616013e0d3618d40022 cde34 2 0 0 ss 2013-4-15 18:20:49 abc34 16F37E59FE494AC9B17A2340387335EF 69
70 402840713e0d3616013e0d3618d30021 cde33 2 0 0 ss 2013-4-15 18:20:49 abc33 16F37E59FE494AC9B17A2340387335EF 70
71 402840713e0d3616013e0d3618d30020 cde32 2 0 0 ss 2013-4-15 18:20:49 abc32 16F37E59FE494AC9B17A2340387335EF 71
72 402840713e0d3616013e0d3618d3001f cde31 2 0 0 ss 2013-4-15 18:20:49 abc31 16F37E59FE494AC9B17A2340387335EF 72
73 402840713e0d3616013e0d3618d3001e cde30 2 0 0 ss 2013-4-15 18:20:49 abc30 16F37E59FE494AC9B17A2340387335EF 73
74 402840713e0d3616013e0d3618d3001d cde29 2 0 0 ss 2013-4-15 18:20:49 abc29 16F37E59FE494AC9B17A2340387335EF 74
75 402840713e0d3616013e0d3618d3001c cde28 2 0 0 ss 2013-4-15 18:20:49 abc28 16F37E59FE494AC9B17A2340387335EF 75
76 402840713e0d3616013e0d3618d3001b cde27 2 0 0 ss 2013-4-15 18:20:49 abc27 16F37E59FE494AC9B17A2340387335EF 76
77 402840713e0d3616013e0d3618d3001a cde26 2 0 0 ss 2013-4-15 18:20:49 abc26 16F37E59FE494AC9B17A2340387335EF 77
78 402840713e0d3616013e0d3618d30019 cde25 2 0 0 ss 2013-4-15 18:20:49 abc25 16F37E59FE494AC9B17A2340387335EF 78
79 402840713e0d3616013e0d3618d30018 cde24 2 0 0 ss 2013-4-15 18:20:49 abc24 16F37E59FE494AC9B17A2340387335EF 79
80 402840713e0d3616013e0d3618d20017 cde23 2 0 0 ss 2013-4-15 18:20:49 abc23 16F37E59FE494AC9B17A2340387335EF 80
81 402840713e0d3616013e0d3618d20016 cde22 2 0 0 ss 2013-4-15 18:20:49 abc22 16F37E59FE494AC9B17A2340387335EF 81
82 402840713e0d3616013e0d3618d20015 cde21 2 0 0 ss 2013-4-15 18:20:49 abc21 16F37E59FE494AC9B17A2340387335EF 82
83 402840713e0d3616013e0d3618d20014 cde20 2 0 0 ss 2013-4-15 18:20:49 abc20 16F37E59FE494AC9B17A2340387335EF 83
84 402840713e0d3616013e0d3618d20013 cde19 2 0 0 ss 2013-4-15 18:20:49 abc19 16F37E59FE494AC9B17A2340387335EF 84
85 402840713e0d3616013e0d3618d20012 cde18 2 0 0 ss 2013-4-15 18:20:49 abc18 16F37E59FE494AC9B17A2340387335EF 85
86 402840713e0d3616013e0d3618d20011 cde17 2 0 0 ss 2013-4-15 18:20:49 abc17 16F37E59FE494AC9B17A2340387335EF 86
87 402840713e0d3616013e0d3618d20010 cde16 2 0 0 ss 2013-4-15 18:20:49 abc16 16F37E59FE494AC9B17A2340387335EF 87
88 402840713e0d3616013e0d3618d0000f cde15 2 0 0 ss 2013-4-15 18:20:49 abc15 16F37E59FE494AC9B17A2340387335EF 88
89 402840713e0d3616013e0d3618cf000e cde14 2 0 0 ss 2013-4-15 18:20:49 abc14 16F37E59FE494AC9B17A2340387335EF 89
90 402840713e0d3616013e0d3618cf000d cde13 2 0 0 ss 2013-4-15 18:20:49 abc13 16F37E59FE494AC9B17A2340387335EF 90
91 402840713e0d3616013e0d3618cf000c cde12 2 0 0 ss 2013-4-15 18:20:49 abc12 16F37E59FE494AC9B17A2340387335EF 91
92 402840713e0d3616013e0d3618cf000b cde11 2 0 0 ss 2013-4-15 18:20:49 abc11 16F37E59FE494AC9B17A2340387335EF 92
93 402840713e0d3616013e0d3618cf000a cde10 2 0 0 ss 2013-4-15 18:20:49 abc10 16F37E59FE494AC9B17A2340387335EF 93
94 402840713e0d3616013e0d3618ce0009 cde9 2 0 0 ss 2013-4-15 18:20:49 abc9 16F37E59FE494AC9B17A2340387335EF 94
95 402840713e0d3616013e0d3618ce0008 cde8 2 0 0 ss 2013-4-15 18:20:49 abc8 16F37E59FE494AC9B17A2340387335EF 95
96 402840713e0d3616013e0d3618ce0007 cde7 2 0 0 ss 2013-4-15 18:20:49 abc7 16F37E59FE494AC9B17A2340387335EF 96
97 402840713e0d3616013e0d3618ce0006 cde6 2 0 0 ss 2013-4-15 18:20:49 abc6 16F37E59FE494AC9B17A2340387335EF 97
98 402840713e0d3616013e0d3618ce0005 cde5 2 0 0 ss 2013-4-15 18:20:49 abc5 16F37E59FE494AC9B17A2340387335EF 98
99 402840713e0d3616013e0d3618ce0004 cde4 2 0 0 ss 2013-4-15 18:20:49 abc4 16F37E59FE494AC9B17A2340387335EF 99
100 402840713e0d3616013e0d3618ce0003 cde3 2 0 0 ss 2013-4-15 18:20:49 abc3 16F37E59FE494AC9B17A2340387335EF 100
101 402840713e0d3616013e0d3618ce0002 cde2 2 0 0 ss 2013-4-15 18:20:49 abc2 16F37E59FE494AC9B17A2340387335EF 101
102 402840713e0d3616013e0d3618cd0001 cde1 2 0 0 ss 2013-4-15 18:20:49 abc1 16F37E59FE494AC9B17A2340387335EF 102
103 402840713e0cb39c013e0cb41c040000 dfsdf 2 1 0 ss 2013-4-15 15:58:50 sdfsdfsdf 16F37E59FE494AC9B17A2340387335EF 103
104 402840713e0c9424013e0c94a8c30000 fffff 2 1 0 ss 2013-4-15 15:24:29 dgfdgfdg 16F37E59FE494AC9B17A2340387335EF 104
105 402840713e0c8e00013e0c8f1d190000 asf 2 2 0 ss 2013-4-15 15:18:25 dsfds 16F37E59FE494AC9B17A2340387335EF 105
106 402840713e0c80ce013e0c8158630000 ccc 2 1 0 ss 2013-4-15 15:03:23 dfsd 16F37E59FE494AC9B17A2340387335EF 106
107 402840713e0c5a97013e0c5b0c200000 hhhhh 2 2 0 ss 2013-4-15 14:21:33 fds 16F37E59FE494AC9B17A2340387335EF 107
108 402840713e0c58cd013e0c592b550000 hff 2 0 0 ss 2013-4-15 14:19:30 sdfs 16F37E59FE494AC9B17A2340387335EF 108
109 402840713e0c272d013e0c2e12c40000 gsfsdf 2 0 0 ss 2013-4-15 13:32:26 fdsfds 16F37E59FE494AC9B17A2340387335EF 109
110 402840713e0c25ef013e0c265f720000 fdsf 2 0 0 ss 2013-4-15 13:24:01 fsdfsdf 16F37E59FE494AC9B17A2340387335EF 110
111 402840713e0c2237013e0c228d4b0000 fdsf 2 0 0 ss 2013-4-15 13:19:51 sdfs 16F37E59FE494AC9B17A2340387335EF 111
112 402840713e0c1cc9013e0c1d0dc30000 fsdf 2 0 0 ss 2013-4-15 13:13:50 sdfdsfds 16F37E59FE494AC9B17A2340387335EF 112
113 402840713e0c1be8013e0c1c33bf0000 fsfds 2 0 0 ss 2013-4-15 13:12:54 fdsfsd 16F37E59FE494AC9B17A2340387335EF 113
114 402840713e0c1b0b013e0c1b72320000 fsfsd 2 0 0 ss 2013-4-15 13:12:05 fsdfsd 16F37E59FE494AC9B17A2340387335EF 114
115 402840713e0c19a9013e0c19f49b0000 GDFG 2 0 0 ss 2013-4-15 13:10:27 FDGDFGDF 16F37E59FE494AC9B17A2340387335EF 115
116 402840713e0c17c7013e0c184b200000 fdsfsd 2 0 0 ss 2013-4-15 13:08:38 fsdfsdfs 16F37E59FE494AC9B17A2340387335EF 116
117 402840713e0c0edc013e0c0f79d20000 fsdfds 2 0 0 ss 2013-4-15 12:59:00 fdsfdsfdsfd 16F37E59FE494AC9B17A2340387335EF 117
118 402840713e0bee4f013e0bef28150000 fdsfsd 2 0 0 ss 2013-4-15 12:23:42 fdsfsdf 16F37E59FE494AC9B17A2340387335EF 118
119 402840713e0b92fc013e0b9418cf0000 fdsfdsf 2 0 0 ss 2013-4-15 10:44:15 dsfdsfdsfds 16F37E59FE494AC9B17A2340387335EF 119
but when i code
select row_.*, rownum rownum_ from (select topic0_.id as id6_, topic0_.title as title6_, topic0_.publish_author_id as publish3_6_, topic0_.reply_number as reply4_6_, topic0_.read_number as read5_6_, topic0_.publish_author_name as publish6_6_, topic0_.publish_time as publish7_6_, topic0_.last_reply_time as last8_6_, topic0_.last_reply_name as last9_6_, topic0_.main_text as main10_6_, topic0_.FORUM_ID as FORUM11_6_ from topic topic0_ where topic0_.FORUM_ID='16F37E59FE494AC9B17A2340387335EF' order by case when topic0_.last_reply_time is not null then topic0_.last_reply_time else topic0_.publish_time end desc) row_ where rownum <=6;
the result is
ID6_ TITLE6_ PUBLISH3_6_ REPLY4_6_ READ5_6_ PUBLISH6_6_ PUBLISH7_6_ LAST8_6_ LAST9_6_ MAIN10_6_ FORUM11_6_ ROWNUM_
1 402840713e108ea4013e1090c95c0001 trytr 2 2 0 ss 2013-4-16 9:58:44 2013-4-16 9:59:21 ss fdsfds 16F37E59FE494AC9B17A2340387335EF 1
2 402840713e108ea4013e1090127e0000 trytr 2 0 0 ss 2013-4-16 9:57:57 ytrytr 16F37E59FE494AC9B17A2340387335EF 2
3 402840713e0d3616013e0d3618c20000 cde0 2 0 0 ss 2013-4-15 18:20:49 abc0 16F37E59FE494AC9B17A2340387335EF 3
4 402840713e0d3616013e0d3618ce0003 cde3 2 0 0 ss 2013-4-15 18:20:49 abc3 16F37E59FE494AC9B17A2340387335EF 4
5 402840713e0d3616013e0d3618ce0002 cde2 2 0 0 ss 2013-4-15 18:20:49 abc2 16F37E59FE494AC9B17A2340387335EF 5
6 402840713e0d3616013e0d3618cd0001 cde1 2 0 0 ss 2013-4-15 18:20:49 abc1 16F37E59FE494AC9B17A2340387335EF 6
why ? i want the result is:
ID6_ TITLE6_ PUBLISH3_6_ REPLY4_6_ READ5_6_ PUBLISH6_6_ PUBLISH7_6_ LAST8_6_ LAST9_6_ MAIN10_6_ FORUM11_6_ ROWNUM_
1 402840713e108ea4013e1090c95c0001 trytr 2 2 0 ss 2013-4-16 9:58:44 2013-4-16 9:59:21 ss fdsfds 16F37E59FE494AC9B17A2340387335EF 1
2 402840713e108ea4013e1090127e0000 trytr 2 0 0 ss 2013-4-16 9:57:57 ytrytr 16F37E59FE494AC9B17A2340387335EF 2
3 402840713e0d3616013e0d3618c20000 cde0 2 0 0 ss 2013-4-15 18:20:49 abc0 16F37E59FE494AC9B17A2340387335EF 3
4 402840713e0d3616013e0d3618da0063 cde99 2 0 0 ss 2013-4-15 18:20:49 abc99 16F37E59FE494AC9B17A2340387335EF 4
5 402840713e0d3616013e0d3618da0062 cde98 2 0 0 ss 2013-4-15 18:20:49 abc98 16F37E59FE494AC9B17A2340387335EF 5
6 402840713e0d3616013e0d3618da0061 cde97 2 0 0 ss 2013-4-15 18:20:49 abc97 16F37E59FE494AC9B17A2340387335EF 6
If somebody could explain to me I'd be most grateful.

In your first query you have:
select row_.*, rownum rownum_ from (select ... order by ...) row_;
In the second you have:
select row_.*, rownum rownum_ from (select ... order by ...) row_
where rownum <= 6;
rownum is a pseudo-column that is generated as the last step in the query, over the result set that has already been produced. The where rownum <= 6 is being applied against the results of the inner select, but at that point in the outer select you have no explicit order by clause so the order the rownum will be assigned isn't guaranteed.
You could either repeat the order by clause in the outer query, which would make it even more unpleasant to deal with, or more simply move the rownum inside the inner select:
select * from (select ..., rownum rownum_ from ... order by ...)
where rownum_ <= 6;
It's unclear if the results form the your first query are in the correct order anyway; probably, but again probably not guaranteed, so you can also order that to make sure:
select * from (select ..., rownum rownum_ from ... order by ...)
where rownum_ <= 6
order by rownum_;
(Another approach would be to use an analytic function like dense_rank() against the case you're using in the order by, and then use that in the outer where to restrict the number of rows, but that's getting a bit away from the question...)

Oracle doesn't guarantee the same order of rows while executing the same query without order by clause.

The documentation says: ROWNUM is assigned by Oracle to show "the order in which Oracle selects the row from a table or set of joined rows". That is, before the result set is sorted. Find out more.
So if you want to guarantee that the top-n rows are always same rows for a given result set you need to pre-sort it.
select t2.* from
( select t1.*, rownum as rownum_ from
( select t.* from t order by t.whatever ) t1
)
where rownum_ <= 6
;
There are other ways of solving top-n questions - such as using RANK or ROW_NUMBER - but they all rely on an inner query. Here is the above solution done with ROW_NUMBER() instead.
select t1.* from
( select t.*
, row_number() over ( order by t.whatever ) as rownum_
from t ) t1
where rownum_ <= 6
;

Resources