I am using elasticsearch 5.6.13 version, I need some experts configurations for the elasticsearch. I have 3 nodes in the same system (node1,node2,node3) where node1 is master and else 2 data nodes. I have number of indexes around 40, I created all these indexes with default 5 primary shards and some of them have 2 replicas.
What I am facing the issue right now, My data (scraping) is growing day by day and I have 400GB of the data in my one of index. similarly 3 other indexes are also very loaded.
From some last days I am facing the issue while insertion of data my elasticsearch hangs and then the service is killed which effect my processing. I have tried several things. I am sharing the system specs and current ES configuration + logs. Please suggest some solution.
The System Specs:
RAM: 160 GB,
CPU: AMD EPYC 7702P 64-Core Processor,
Drive: 2 TB SSD (The drive in which the ES installed still have 500 GB left)
ES Configuration JVM options:
-Xms26g,
-Xmx26g
(I just try this but not sure what is the perfect heap size for my scenario)
I just edit this above lines and the rest of the file is as defult. I edit this on all three nodes jvm.options files.
ES LOGS
[2021-09-22T12:05:17,983][WARN ][o.e.m.j.JvmGcMonitorService] [sashanode1] [gc][170] overhead, spent [7.1s] collecting in the last [7.2s]
[2021-09-22T12:05:21,868][WARN ][o.e.m.j.JvmGcMonitorService] [sashanode1] [gc][171] overhead, spent [3.7s] collecting in the last [1.9s]
[2021-09-22T12:05:51,190][WARN ][o.e.m.j.JvmGcMonitorService] [sashanode1] [gc][172] overhead, spent [27.7s] collecting in the last [23.3s]
[2021-09-22T12:06:54,629][WARN ][o.e.m.j.JvmGcMonitorService] [cluster_name] [gc][173] overhead, spent [57.5s] collecting in the last [1.1m]
[2021-09-22T12:06:56,536][WARN ][o.e.m.j.JvmGcMonitorService] [cluster_name] [gc][174] overhead, spent [1.9s] collecting in the last [1.9s]
[2021-09-22T12:07:02,176][WARN ][o.e.m.j.JvmGcMonitorService] [cluster_name] [gc][175] overhead, spent [5.4s] collecting in the last [5.6s]
[2021-09-22T12:06:56,546][ERROR][o.e.i.e.Engine ] [cluster_name] [index_name][3] merge failed
java.lang.OutOfMemoryError: Java heap space
[2021-09-22T12:06:56,548][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [cluster_name] fatal error in thread [elasticsearch[cluster_name][bulk][T#25]], exiting
java.lang.OutOfMemoryError: Java heap space
Some more logs
[2021-09-22T12:10:06,526][INFO ][o.e.n.Node ] [cluster_name] initializing ...
[2021-09-22T12:10:06,589][INFO ][o.e.e.NodeEnvironment ] [cluster_name] using [1] data paths, mounts [[(D:)]], net usable_space [563.3gb], net total_space [1.7tb], spins? [unknown], types [NTFS]
[2021-09-22T12:10:06,589][INFO ][o.e.e.NodeEnvironment ] [cluster_name] heap size [1.9gb], compressed ordinary object pointers [true]
[2021-09-22T12:10:07,239][INFO ][o.e.n.Node ] [cluster_name] node name [sashanode1], node ID [2p-ux-OXRKGuxmN0efvF9Q]
[2021-09-22T12:10:07,240][INFO ][o.e.n.Node ] [cluster_name] version[5.6.13], pid[57096], build[4d5320b/2018-10-30T19:05:08.237Z], OS[Windows Server 2019/10.0/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_261/25.261-b12]
[2021-09-22T12:10:07,240][INFO ][o.e.n.Node ] [cluster_name] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Delasticsearch, -Des.path.home=D:\Databases\ES\elastic and kibana 5.6.13\es_node_1, -Des.default.path.logs=D:\Databases\ES\elastic and kibana 5.6.13\es_node_1\logs, -Des.default.path.data=D:\Databases\ES\elastic and kibana 5.6.13\es_node_1\data, -Des.default.path.conf=D:\Databases\ES\elastic and kibana 5.6.13\es_node_1\config, exit, -Xms2048m, -Xmx2048m, -Xss1024k]
Also in my ES folder there are so many files with the random names (java_pid197036.hprof)
Further details can be shared please suggest any further configurations.
Thanks
The output for _cluster/stats?pretty&human is
{ "_nodes": { "total": 3, "successful": 3, "failed": 0 }, "cluster_name": "cluster_name", "timestamp": 1632375228033, "status": "red", "indices": { "count": 42, "shards": { "total": 508, "primaries": 217, "replication": 1.3410138248847927, "index": { "shards": { "min": 2, "max": 60, "avg": 12.095238095238095 }, "primaries": { "min": 1, "max": 20, "avg": 5.166666666666667 }, "replication": { "min": 1.0, "max": 2.0, "avg": 1.2857142857142858 } } }, "docs": { "count": 107283077, "deleted": 1047418 }, "store": { "size": "530.2gb", "size_in_bytes": 569385384976, "throttle_time": "0s", "throttle_time_in_millis": 0 }, "fielddata": { "memory_size": "0b", "memory_size_in_bytes": 0, "evictions": 0 }, "query_cache": { "memory_size": "0b", "memory_size_in_bytes": 0, "total_count": 0, "hit_count": 0, "miss_count": 0, "cache_size": 0, "cache_count": 0, "evictions": 0 }, "completion": { "size": "0b", "size_in_bytes": 0 }, "segments": { "count": 3781, "memory": "2gb", "memory_in_bytes": 2174286255, "terms_memory": "1.7gb", "terms_memory_in_bytes": 1863786029, "stored_fields_memory": "105.6mb", "stored_fields_memory_in_bytes": 110789048, "term_vectors_memory": "0b", "term_vectors_memory_in_bytes": 0, "norms_memory": "31.9mb", "norms_memory_in_bytes": 33527808, "points_memory": "13.1mb", "points_memory_in_bytes": 13742470, "doc_values_memory": "145.3mb", "doc_values_memory_in_bytes": 152440900, "index_writer_memory": "0b", "index_writer_memory_in_bytes": 0, "version_map_memory": "0b", "version_map_memory_in_bytes": 0, "fixed_bit_set": "0b", "fixed_bit_set_memory_in_bytes": 0, "max_unsafe_auto_id_timestamp": 1632340789677, "file_sizes": { } } }, "nodes": { "count": { "total": 3, "data": 3, "coordinating_only": 0, "master": 1, "ingest": 3 }, "versions": [ "5.6.13" ], "os": { "available_processors": 192, "allocated_processors": 96, "names": [ { "name": "Windows Server 2019", "count": 3 } ], "mem": { "total": "478.4gb", "total_in_bytes": 513717497856, "free": "119.7gb", "free_in_bytes": 128535437312, "used": "358.7gb", "used_in_bytes": 385182060544, "free_percent": 25, "used_percent": 75 } }, "process": { "cpu": { "percent": 5 }, "open_file_descriptors": { "min": -1, "max": -1, "avg": 0 } }, "jvm": { "max_uptime": "1.9d", "max_uptime_in_millis": 167165106, "versions": [ { "version": "1.8.0_261", "vm_name": "Java HotSpot(TM) 64-Bit Server VM", "vm_version": "25.261-b12", "vm_vendor": "Oracle Corporation", "count": 3 } ], "mem": { "heap_used": "5gb", "heap_used_in_bytes": 5460944144, "heap_max": "5.8gb", "heap_max_in_bytes": 6227755008 }, "threads": 835 }, "fs": { "total": "1.7tb", "total_in_bytes": 1920365228032, "free": "499.1gb", "free_in_bytes": 535939969024, "available": "499.1gb", "available_in_bytes": 535939969024 }, "plugins": [ ], "network_types": { "transport_types": { "netty4": 3 }, "http_types": { "netty4": 3 } } } }
The jvm.options file.
## JVM configuration
################################################################
## IMPORTANT: JVM heap size
################################################################
##
## You should always set the min and max JVM heap
## size to the same value. For example, to set
## the heap to 4 GB, set:
##
## -Xms4g
## -Xmx4g
##
## See https://www.elastic.co/guide/en/elasticsearch/reference/current/heap-size.html
## for more information
##
################################################################
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms26g
-Xmx26g
################################################################
## Expert settings
################################################################
##
## All settings below this section are considered
## expert settings. Don't tamper with them unless
## you understand what you are doing
##
################################################################
## GC configuration
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
## optimizations
# pre-touch memory pages used by the JVM during initialization
-XX:+AlwaysPreTouch
## basic
# force the server VM (remove on 32-bit client JVMs)
-server
# explicitly set the stack size (reduce to 320k on 32-bit client JVMs)
-Xss1m
# set to headless, just in case
-Djava.awt.headless=true
# ensure UTF-8 encoding by default (e.g. filenames)
-Dfile.encoding=UTF-8
# use our provided JNA always versus the system one
-Djna.nosys=true
# use old-style file permissions on JDK9
-Djdk.io.permissionsUseCanonicalPath=true
# flags to configure Netty
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
# log4j 2
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Dlog4j.skipJansi=true
## heap dumps
# generate a heap dump when an allocation from the Java heap fails
# heap dumps are created in the working directory of the JVM
-XX:+HeapDumpOnOutOfMemoryError
# specify an alternative path for heap dumps
# ensure the directory exists and has sufficient space
#-XX:HeapDumpPath=${heap.dump.path}
## GC logging
#-XX:+PrintGCDetails
#-XX:+PrintGCTimeStamps
#-XX:+PrintGCDateStamps
#-XX:+PrintClassHistogram
#-XX:+PrintTenuringDistribution
#-XX:+PrintGCApplicationStoppedTime
# log GC status to a file with time stamps
# ensure the directory exists
#-Xloggc:${loggc}
# By default, the GC log file will not rotate.
# By uncommenting the lines below, the GC log file
# will be rotated every 128MB at most 32 times.
#-XX:+UseGCLogFileRotation
#-XX:NumberOfGCLogFiles=32
#-XX:GCLogFileSize=128M
# Elasticsearch 5.0.0 will throw an exception on unquoted field names in JSON.
# If documents were already indexed with unquoted fields in a previous version
# of Elasticsearch, some operations may throw errors.
#
# WARNING: This option will be removed in Elasticsearch 6.0.0 and is provided
# only for migration purposes.
#-Delasticsearch.json.allow_unquoted_field_names=true
and the elasticsearch.yml (master node)
cluster.name: cluster_name
node.name: node1
node.master : true
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["192.168.11.159", "192.168.11.157"]
My issue is solved, It is due to the heap size issue, actually I am running the ES as service and the heap size is by default 2 GB and it is not reflecting.
I just install the new service with the updated options.jvm file with heap size of 10 GB, and then run my cluster. It reflect the heap size from 2 GB to 10 GB.
And my problem is solved. Thanks for the suggestions.
to check your heap size use this command.
http://localhost:9200/_cat/nodes?h=heap*&v
I'm investigating latency between API GW and Lambda. Although the latency may seem low, there is a source of latency that I cannot discern.
It appears to be integration latency from when API GW makes a request to Lambda, to when Lambda begins execution.
We have ruled out cold start by enabling provisioned concurrency and have no spill over into un-provisioned containers.
Example Access log & Corresponding Lambda
{
"requestTime": "09/Jul/2021:12:18:48 +0000",
"requestId": "redacted",
"httpMethod": "POST",
"path": "/redacted",
"resourcePath": "/redacted",
"status": 200,
"responseLatency": 56,
"wafError": "-",
"wafStatus": "-",
"wafLatency": "-",
"authnError": "-",
"authnStatus": "200",
"authnLatency": "2",
"authzError": "-",
"authzStatus": "200",
"authzLatency": "0",
"integrationRequestId": "redacted",
"integrationResponseStatus": "200",
"integrationLatency": "52",
"integrationServiceStatus": "200",
"identitySourceIp": "redacted",
"identityUserAgent": "redacted",
"identityUser": "redacted"
}
REPORT RequestId: redacted Duration: 37.83 ms Billed Duration: 38 ms Memory Size: 2048 MB Max Memory Used: 77 MB
Has anyone encountered this gap before? The overhead from API GW seems minimal.
We use seaweedfs 1.78
When i use grpc delete a file via filer.
curl -X DELETE http://filer1:9889/dataset/qiantao/1.txt
It return success.
Because I have 10 filer. after delete!
curl -H "Accept: application/json" "http://filer2:9889/dataset/qiantao/?pretty=y" |grep qiantao |grep txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 15723 0 15723 0 0 1917k 0 --:--:-- --:--:-- --:--:-- 2193k
"FullPath": "/dataset/qiantao/1.txt",
If I start a new filer.
It can not got /dataset/qiantao/1.txt; It looks perfect!!!!
But in exist filers.
Filer get file info below.
curl -H "Accept: application/json" "http://filer1:9889/dataset/qiantao/?pretty=y&limit=1"
{
"Path": "/dataset/qiantao",
"Entries": [
{
"FullPath": "/dataset/qiantao/1.txt",
"Mtime": "2020-12-07T11:15:59+08:00",
"Crtime": "2020-12-07T11:15:59+08:00",
"Mode": 432,
"Uid": 0,
"Gid": 0,
"Mime": "text/plain",
"Replication": "010",
"Collection": "",
"TtlSec": 0,
"UserName": "",
"GroupNames": null,
"SymlinkTarget": "",
"Md5": null,
"Extended": null,
"chunks": [
{
"file_id": "4328,587fb084df9f9dbf",
"size": 2,
"mtime": 1607310959158810676,
"e_tag": "c7c83966",
"fid": {
"volume_id": 4328,
"file_key": 1484763268,
"cookie": 3751779775
}
}
]
}
],
"Limit": 1,
"LastFileName": "1.txt",
"ShouldDisplayLoadMore": true
Get volume info below.
{
"Id": 4328,
"Size": 31492542356,
"ReplicaPlacement": {
"SameRackCount": 0,
"DiffRackCount": 1,
"DiffDataCenterCount": 0
},
"Ttl": {
"Count": 0,
"Unit": 0
},
"Collection": "",
"Version": 3,
"FileCount": 111030,
"DeleteCount": 709,
"DeletedByteCount": 1628822733,
"ReadOnly": false,
"CompactRevision": 0,
"ModifiedAtSecond": 0,
"RemoteStorageName": "",
"RemoteStorageKey": ""
},
So download 4328.idx from volume server. and use see_idx lookup it.
./see_idx -dir /Users/qiantao/Documents/seaweedfs -volumeId=4328 -v=4 |grep 587fb084
key:587fb084 offset:2802901546 size:57
key:587fb084 offset:3937021600 size:4294967295
It looks like key:587fb084 is covered with new?
So How can I fix this problem to make it appear normal?
4294967295 is a tombstone, marking the entry has been deleted.
I'm trying to create a table or data grid with dynamic row and column count in my NativeScript application. I have some product categories and some products in these categories. Products have some specifications with respect to its belonging category. Here is an example JSON for products in graphic cards category:
{
"parts": [
{
"id": 1,
"name": "GTX 1080",
"stockCount": 10,
"specifications": [
{
"id": 1,
"name": "Memory",
"value": "11 GB"
},
{
"id": 2,
"name": "Core Clock",
"value": "1500 MHz"
}
]
},
{
"id": 2,
"name": "RX 580",
"stockCount": 8,
"specifications": [
{
"id": 1,
"name": "Memory",
"value": "8 GB"
},
{
"id": 2,
"name": "Core Clock",
"value": "1366 MHz"
}
]
}
]
}
Here is another example in CPU category:
{
"parts": [
{
"id": 3,
"name": "i5 7600K",
"stockCount": 8,
"specifications": [
{
"id": 3,
"name": "Socket",
"value": "LGA 1151"
},
{
"id": 4,
"name": "Frequency",
"value": "3.8 GHz"
},
{
"id": 5,
"name": "L3 Cache",
"value": "6 MB"
}
]
},
{
"id": 4,
"name": "Ryzen 7 1700",
"stockCount": 15,
"specifications": [
{
"id": 3,
"name": "Socket",
"value": "AM4"
},
{
"id": 4,
"name": "Frequency",
"value": "3.0 GHz"
},
{
"id": 5,
"name": "L3 Cache",
"value": "16MB"
}
]
}
]
}
I want to show graphic cards as a table like this:
Name Stock Memory Core Clock
GTX 1080 10 11 GB 1500 MHz
RX 580 8 8 GB 1366 MHz
and CPUs like this:
Name Stock Socket Frequency L3 Cache
i5 7600K 8 LGA 1151 3.8 GHz 6 MB
Ryzen 7 1700 15 AM4 3.0 GHz 16 MB
I have tried RadListView with GridLayout but cannot do this. If specification count for all categories would be constant, I could easily create GridLayout with constant number of columns like this:
<GridLayout columns="*, auto, auto, auto, auto">...</GridLayout>
But arbitrary number of specifications put me into a challenge here.
Is there some way in NativeScript Angular to achieve this? I'm using 4.1.0 version of NativeScript.
you can generate value dynamically and then bind it to GridLayout cloumns property something like this
<GridLayout [columns]="genCols(item)">
&
genCols(item){
if(item.columns){
return item.columns;
}else{
item.columns="*, auto";
item.specifications.forEach((el)=>{
item.columns+=",auto";
})
return item.columns
}
}
if users can add specifications and specification count for each item is same then simpler way will be to use single variable and set its value on ngOnInit and update it on specification added like columns=genCol(items[0]) and then <GridLayout [columns]="columns">
For those looking for a solution strictly in the template, I used String.prototype.repeat() within the [columns] attribute:
[columns]="'*, auto' + ', auto'.repeat(parts.specifications.length)"
Full example:
<GridLayout [colums]="'*, auto, ' + 'auto, '.repeat(parts.specifications.length)">
<Label col="0" [text]="parts.name"></Label>
<Label col="1" [text]="parts.stockCount"></Label>
<Label [col]="i" [text]="spec.value" *ngFor="let spec of parts.specifications; let i=index+2"></Label>
</GridLayout>
This morning, we were alerted that a few machines in a 4 node Elasticsearch clusters are running low on disk space. We're using 5 shards, with single replication.
The status report shows index sizes that line up with the disk usage we're seeing. However, the confusing thing, is that the replicated shards are very out of line for shards 2 and 4. I understand that shard sizes can vary between replicas; however the size differences we are seeing are enormous:
"shards": {
...
"2": [
{
"routing": {
"state": "STARTED",
"primary": true,
"node": "LV__Sh-vToyTcuuxwnZaAg",
"relocating_node": null,
"shard": 2,
"index": "eventdata"
},
"state": "STARTED",
"index": {
"size_in_bytes": 87706293809
},
......
},
{
"routing": {
"state": "STARTED",
"primary": false,
"node": "U7eYVll7ToWS9lPkyhql6g",
"relocating_node": null,
"shard": 2,
"index": "eventdata"
},
"state": "STARTED",
"index": {
"size_in_bytes": 42652984946
},
Some interesting data points:
There's a lot of merging happening on the cluster at the moment. Could this be a factor? If merges make a copy of the index prior to merging, then that would explain a lot.
The bigger shard is almost exactly twice the size of the the smaller shard. coincidence? (I think not).
Why are we seeing indices at double the size of their replicas in our cluster? merging? some other reason?