stream load balancing is not working on loki ingester - grafana-loki

Question

I'm beginner to loki, and I need some help.
I am running loki-distributed on EKS.
chart version : 0.43.0
loki version : 4.2.4
Log flow is : logstash(4 node) is consuming log from kafka and push the log to Loki's domain. loki domain is connected from AWS ALB ingress
I have 4 ingester pod, and ring status is normal.
image
I thought distributor do load balancing for 4 ingester.
but only 2 ingester was used. and they take too much memory, and oom killed and relaunch. It is never ending..
image
why 2 ingester is idle and never use??
my configuration something wrong?
this is my configuration. could you help me please??
config: |
auth_enabled: false
server:
http_listen_port: 3100
grpc_server_min_time_between_pings: 10s
grpc_server_ping_without_stream_allowed: true
grpc_server_max_recv_msg_size: 104857600
grpc_server_max_send_msg_size: 104857600
distributor:
ring:
kvstore:
store: memberlist
heartbeat_timeout: 30s
memberlist:
join_members:
- loki-memberlist
ingester:
lifecycler:
join_after: 0s
ring:
kvstore:
store: memberlist
replication_factor: 1
chunk_idle_period: 1h
chunk_target_size: 1536000
chunk_block_size: 262144
chunk_encoding: snappy
chunk_retain_period: 1m
max_transfer_retries: 0
autoforget_unhealthy: false
wal:
dir: /var/loki/wal
limits_config:
enforce_metric_name: false
reject_old_samples: true
reject_old_samples_max_age: 168h
max_cache_freshness_per_query: 10m
max_streams_per_user: 0
max_query_length: 720h
max_query_parallelism: 24
max_entries_limit_per_query: 10000
ingestion_burst_size_mb: 32
ingestion_rate_mb: 16
cardinality_limit: 1000000
schema_config:
configs:
- from: "2021-12-24"
store: aws
object_store: s3
schema: v11
index:
prefix: {{ index_name }}
period: 720h
storage_config:
aws:
s3: s3://ap-northeast-2/{{ bucket_name }}
dynamodb:
dynamodb_url: dynamodb://ap-northeast-2
http_config:
response_header_timeout: 5s
boltdb_shipper:
shared_store: s3
active_index_directory: /var/loki/index
cache_location: /var/loki/cache
cache_ttl: 168h
index_gateway_client:
server_address: dns://loki-index-gateway:9095
index_cache_validity: 168h
index_queries_cache_config:
enable_fifocache: true
default_validity: 168h
fifocache:
validity: 168h
chunk_store_config:
max_look_back_period : 0s
chunk_cache_config:
enable_fifocache: true
default_validity: 168h
fifocache:
validity: 168h
table_manager:
retention_deletes_enabled: false
throughput_updates_disabled: false
retention_period: 0
chunk_tables_provisioning:
enable_ondemand_throughput_mode: true
enable_inactive_throughput_on_demand_mode: true
provisioned_write_throughput: 0
provisioned_read_throughput: 0
inactive_write_throughput: 0
inactive_read_throughput: 0
index_tables_provisioning:
enable_ondemand_throughput_mode: true
enable_inactive_throughput_on_demand_mode: true
provisioned_write_throughput: 0
provisioned_read_throughput: 0
inactive_write_throughput: 0
inactive_read_throughput: 0
querier:
query_timeout: 5m
query_ingesters_within: 1h
engine:
timeout: 5m
query_range:
align_queries_with_step: true
max_retries: 5
split_queries_by_interval: 10m
cache_results: true
align_queries_with_step: true
parallelise_shardable_queries: true
results_cache:
cache:
enable_fifocache: true
default_validity: 168h
fifocache:
validity: 168h
frontend_worker:
frontend_address: loki-query-frontend:9095
#scheduler_address: loki-scheduler:9095
grpc_client_config:
max_recv_msg_size: 104857600
max_send_msg_size: 104857600
match_max_concurrent: false
parallelism: 8
frontend:
log_queries_longer_than: 1m
compress_responses: true
tail_proxy_url: http://loki-querier:3100
#scheduler_address: loki-scheduler:9095
compactor:
shared_store: filesystem
ruler:
enable_api: true
storage:
type: s3
s3:
s3: s3://ap-northeast-2/{{ rule-bucket-name }}
rule_path: /tmp/loki/scratch
alertmanager_url: http://alertmanager:9093