I watched the site memo about increasing indexing performance.
this is site link
This link instruct me how to increase performance. but, it did not improve indexing speed in elasticsearch when I used to bulk python api with elasticsearch-py.
even all configuration change didn't affect bulk indexing performance.
i used parallel process or thread. max avg 30000 indexing per second.
what did i fault?
master node : 1
data node : 5 include master node
CPU : Intel(R) Xeon(R) CPU E5645 # 2.40GHz
RAM : 32G
ES_HEAPSIZE : 10G
Thanks
It actually increase performance dramatically (more than 50% on my side). You just need to disable refresh_interval (enable it again when you finish indexing data)
curl -XPUT "http://localhost:9200/$INDEX_NAME/_settings" -d '{ "index" : { "refresh_interval" : "-1" }}'
#index data......
curl -XPUT "http://localhost:9200/$INDEX_NAME/_settings" -d '{ "index" : { "refresh_interval" : "1s" }}'
Related
I am using requests library to connect to elasticsearch for fetching data. I have
26 indices,
spread across 2 nodes,
with 1st node having 16GB RAM / 8 vCPU and the
2nd 8GB RAM / 4 vCPU.
All my nodes are in AWS EC2.
In all I have around 200 GB of data. I am primarily using the database for aggregation exercises.
A typical data record would look like this
SITE_ID DATE HOUR MAID
123 2021-05-05 16 m434
I am using the following python3 definition to send the request and get the data.
def elasticsearch_curl(uri, json_body='',verb='get'):
headers={'Content-Type': 'application/json',}
try:
resp = requests.get(uri, headers=headers, data=json_body)
try:
resp_text = json.loads(resp.text)
except:
print("Error")
except Exception as error:
print('\nelasticsearch_curl() error:', error)
return resp_text
##Variables
tabsite : ['r123','r234'] ##names of indices
siteid : [123,124,125] ##name of sites
I am using the following code to get the data:
for key,value in tabsite.items():
k=key.replace('_','')
if es.indices.exists(index=k):
url="http://localhost:9200/"+str(k)+"/_search"
jb1='{"size":0,"query": {"bool" : {"filter" : [{"terms" : {"site_id": ' + str(siteid) + '}},{"range" : {"date" : \
{"gte": "'+str(st)+'","lte": "'+str(ed)+'"}}}]}}, "aggs" : {"group_by" : {"terms": {"field": "site_id","size":100},"aggs" : {"bydate" : {"terms" : \
{"field":"date","size": 10000},"aggs" : {"uv" : {"cardinality": {"field": "maid"}}}}}}}}'
try:
r2=elasticsearch_curl(url, json_body=jb1)
k1=r2.get('aggregations',{}).get('group_by',{}).get('buckets')
print(k1)
except:
pass
The above code returns the data from r123 which has 18GB of data while it fails to get it from r234 which has 55 GB of data.
I am getting the following error:
elasticsearch_curl() error: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
I have tried the following:
Try running the above code in a machine which has only r234 index with around 45GB of data. It worked.
I tried increasing the RAM size of the 2nd machine in production from 8GB to 16GB - it failed.
When I searched for options here, I understood I need to close the headers. I am not sure how.
I have the following questions:
How do I keep my elasticsearch nodes stable without getting them shutdown automatically?
How do I get rid of the above error which shuts down one of the nodes or both.
Is there any optimimal configuration setting ratio for volume of data : number of nodes : amount of RAM / vCPUs.
We recently upgraded our elastic cloud deployment from 6.8.5 to 7.9
After the upgrade we are seeing the following error time to time.
{
"error" : {
"root_cause" : [
{
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [416906520/397.5mb], which is larger than the limit of [408420352/389.5mb], real usage: [416906520/397.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=32399/31.6kb, in_flight_requests=0/0b, model_inference=0/0b, accounting=4714192/4.4mb]",
"bytes_wanted" : 416906520,
"bytes_limit" : 408420352,
"durability" : "PERMANENT"
}
],
"type" : "circuit_breaking_exception",
"reason" : "[parent] Data too large, data for [<http_request>] would be [416906520/397.5mb], which is larger than the limit of [408420352/389.5mb], real usage: [416906520/397.5mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=32399/31.6kb, in_flight_requests=0/0b, model_inference=0/0b, accounting=4714192/4.4mb]",
"bytes_wanted" : 416906520,
"bytes_limit" : 408420352,
"durability" : "PERMANENT"
},
"status" : 429
}
This deployment consists of only one node with 1G memory. We would like to know the cause of this error. Is it due to the upgrade?
Thank you.
First, the circuit breaker is a protection that some request doesn't push your cluster over the limit of what it can handle — this is killing a single request rather than (potentially) the entire cluster. Also note that this HTTP request alone isn't too large, but it trips the parent circuit breaker — so this request on top of everything else would be too much.
The initial circuit breaker was already added in 6.2.0, but was tightened down further in 7.0.0. I assume that's the reason why you are seeing this (more frequently) now.
You could change the indices.breaker.total.limit, but this isn't a magic switch to get more out of your cluster. 1GB of memory might just not be enough for what you are trying to do.
I am trying to run elastic search and using the following command I am trying to put data-
'curl -XPOST http://localhost:9200/_bulk?pretty --data-binary #data_.json'
But I am getting the following error-
"create" : {
"_index" : "appname-docm",
"_type" : "HYD",
"_id" : "AVVYfsk7M5xgvmX8VR_B",
"status" : 429,
"error" : {
"type" : "es_rejected_execution_exception",
"reason" : "rejected execution of org.elasticsearch.transport.TransportService$4#c8998f4 on EsThreadPoolExecutor[bulk, queue capacity = 50, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor#553aee29[Running, pool size = 4, active threads = 4, queued tasks = 50, completed tasks = 0]]"
}
}
},
I tried increasing the queue size by-
threadpool.search.queue_size: 100000
But I still get the same error.
The problem that you are getting is because the bulk operations queue is full.
A node ES has many threads pools, generic, search, index, suggest, bulk, etc.
In your case the problem is due to the queue of bulk operations is full.
Try adjusting the queue size of thread pool of bulk operation:
thread_pool.bulk.queue_size: 100
Or reduce the amount of bulk operations that you are sending at once.
For more details see https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-threadpool.html
Try the following:
curl -XPUT localhost:9200/_cluster/settings -d '{ "transient" : { "threadpool.bulk.queue_size" : 500 } }'
Edit:
And to Get current settings
curl -X GET "localhost:9200/_cluster/settings?include_defaults=true"
I'm currently developing a mobile application powered by a Mongo database, and however everything is working fine right now, we want to add Sharding to be prepared for the future.
In order to test this, we've created a lab environment (running in Hyper-V) to test the various scenario's:
The following servers have been created:
Ubuntu Server 14.04.3 Non-Sharding (Database Server) (256 MB Ram / Limit to 10% CPU).
Ubuntu Server 14.04.3 Sharding (Configuration Server) (256 MB Ram / Limit to 10% CPU).
Ubuntu Server 14.04.3 Sharding (Query Router Server) (256 MB Ram / Limit to 10% CPU).
Ubuntu Server 14.04.3 Sharding (Database Server 01) (256 MB Ram / Limit to 10% CPU).
Ubuntu Server 14.04.3 Sharding (Database Server 02) (256 MB Ram / Limit to 10% CPU).
A small console application have been created in C# to be able to measure the time to perform an insert.
This console application does import 10.000 persons with the following properties:
- Name
- Firstname
- Full Name
- Date Of Birth
- Id
All 10.000 records differs only by '_id', all the other fields are the same for all the records.
It's important to note that every test is exactely run 3 times.
After every test, the database is removed so the system is clean again.
Find the results of the test below:
Insert 10.000 records without sharding
Writing 10.000 records | Non-Sharding environment - Full Disk IO #1: 14 Seconds.
Writing 10.000 records | Non-Sharding environment - Full Disk IO #2: 14 Seconds.
Writing 10.000 records | Non-Sharding environment - Full Disk IO #3: 12 Seconds.
Insert 10.000 records with single database shard
Note: Sharding key has been set to hashed _id field.
See Json below for (partial) sharding information:
shards:
{ "_id" : "shard0000", "host" : "192.168.137.12:27017" }
databases:
{ "_id" : "DemoDatabase", "primary" : "shard0000", "partitioned" : true }
DemoDatabase.persons
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard0000 2
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong(0) } on : shard0000 Timestamp(1, 1)
{ "_id" : NumberLong(0) } -->> { "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 2)
Results:
Writing 10.000 records | Single Sharding environment - Full Disk IO #1: 1 Minute, 59 Seconds.
Writing 10.000 records | Single Sharding environment - Full Disk IO #2: 1 Minute, 51 Seconds.
Writing 10.000 records | Single Sharding environment - Full Disk IO #3: 1 Minute, 52 Seconds.
Insert 10.000 records with double database shard
Note: Sharding key has been set to hashed _id field.
See Json below for (partial) sharding information:
shards:
{ "_id" : "shard0000", "host" : "192.168.137.12:27017" }
{ "_id" : "shard0001", "host" : "192.168.137.13:27017" }
databases:
{ "_id" : "DemoDatabase", "primary" : "shard0000", "partitioned" : true }
DemoDatabase.persons
shard key: { "_id" : "hashed" }
unique: false
balancing: true
chunks:
shard0000 2
{ "_id" : { "$minKey" : 1 } } -->> { "_id" : NumberLong("-4611686018427387902") } on : shard0000 Timestamp(2, 2)
{ "_id" : NumberLong("-4611686018427387902") } -->> { "_id" : NumberLong(0) } on : shard0000 Timestamp(2, 3)
{ "_id" : NumberLong(0) } -->> { "_id" : NumberLong("4611686018427387902") } on : shard0001 Timestamp(2, 4)
{ "_id" : NumberLong("4611686018427387902") } -->> { "_id" : { "$maxKey" : 1 } } on : shard0001 Timestamp(2, 5)
Results:
Writing 10.000 records | Single Sharding environment - Full Disk IO #1: 49 Seconds.
Writing 10.000 records | Single Sharding environment - Full Disk IO #2: 53 Seconds.
Writing 10.000 records | Single Sharding environment - Full Disk IO #3: 54 Seconds.
According to the tests executed above, sharding does work, the more shards that I add, the better the performance.
However, I don't understand why I'm facing such a huge performance drop when working with shards rather than using a single server.
I need to blazing fast reading and writing s I tought that sharding would be the solution, but it seems that I'm missing something here.
Anyone why can point me in the right direction?
Kind regards
The layers between the routing server and config server, routing server and data nodes add latency.
If you have 1ms ping * 10k inserts, you have an 10 seconds of latency that does not appear in the unsharded setup.
Depending on your configured level of write-concern (if you configured any level of write-acknowledgement), you could have an additional 10 seconds to your benchmarks on the sharded environment due to blocking until an acknowledgement is received from the data node.
If your write-concern is set to acknowledge and you have replica nodes, then you also have to wait for the write to propagate to your replica nodes, adding additional network latency. (You don't appear to have replica nodes though). And depending on your network topology, write-concern could add multiple layers of network latency if you use the default setting to allow chained replication (secondaries sync from other secondaries). https://docs.mongodb.org/manual/tutorial/manage-chained-replication/. If you have additional indexes and write concern, each replica node will have to write that index before returning a write-acknowledgement (it is possible to disable indexes on replica nodes though)
Without sharding and without replication (but with write-acknowledgement), while your inserts would still block on the insert, there is no additional latency due to the network layer.
Hashing the _id field also has a cost that accumulates to maybe a few seconds total for 10k. You can use an _id field with a high degree of randomness to avoid hashing, but I don't think this affects performance much.
In diagnosing a high CPU mongodb, we found many slow (6-7 secs) queries. All of those are related to "ns" : "mydb.$cmd".
Slow query entry look like below :
{
"_id" : ObjectId("5571b739f65f7e64bb806362"),
"op" : "command",
"ns" : "mydb.$cmd",
"command" : {
"aggregate" : "MyCollection",
"pipeline" : [
{
"$mergeCursors" : [
{
"host" : "abc:27005",
"id" : NumberLong(82775337156)
}
]
}
]
},
"keyUpdates" : 0,
"numYield" : 0,
"lockStats" : {
"timeLockedMicros" : {
"r" : NumberLong(12),
"w" : NumberLong(0)
},
"timeAcquiringMicros" : {
"r" : NumberLong(2),
"w" : NumberLong(2680)
}
},
"responseLength" : 12312,
"millis" : 6142,
"execStats" : {},
"ts" : ISODate("2015-06-05T12:35:40.801Z"),
"client" : "1.1.1.1",
"allUsers" : [],
"user" : ""
}
We are not sure what part of code causing these queries. How shall we proceed to find / debug what queries from application causing these $cmd slow queries ?
Those logs are actually the queries issued when running a command against the specified database (mydb in your case). This is therefore just some aggregation command being run against your MongoDB.
If your application is not doing this directly, it would appear (as documented in http://dbattish.tumblr.com/post/108652372056/joins-in-mongodb) that the $mergecursors variant is used from v2.6 to merge queries across shards.
My test shows that MongoDB uses always ~90-100% CPU when it deals with concurrent requests. This beacause I move to MySQL. My app with the thame simple queries work 3x faster with MySQL and i uses much less CPU. I will create an artciel soon with full testing. For now, just look to CPU usage of MongoDB and MariaDB for queries with X=5, 10, 25, 50, 100, 500, 1000 concurrent connections.
siege -b -cX -t1M url
As I realized, high CPU usage doesn't related to hight CPU usage. I mean, even very simple queries with concurrent requests make MongoDB to use 100% CPU.
All tests with 1vCPU and 1Gb memory and connection pool size 10
MongoDB
MySQL
I did many tests with different configurations (4vCPU, 6G Memory) and always MongoDB was use more CPU then MySQL. What you can try with MongoDB is:
Change connection loop size. I hope you don't open connection per query.
Are you using Mongoose? Try with Native Nodejs MYSQL Drivers - it much faster.
I very disappointed from MongodDB for reading data. Not only that MySQL uses much less CPU, it was always at least 3x faster!