I'm using solr 1.4 and using QueryElevation Component for guaranteed search position. I have around 700,000 documents with 1 Mb elevation file. It turns out it is quite slow on the newrelic monitoring website:
Slowest Components Count Exclusive Total
QueryElevationComponent 1 506,858 ms 100% 506,858 ms 100%
SolrIndexSearcher 1 2.0 ms 0% 2.0 ms 0%
org.apache.solr.servlet.SolrDispatchFilter.doFilter() 1 1.0 ms 0% 506,862 ms 100%
QueryComponent 1 1.0 ms 0% 1.0 ms 0%
DebugComponent 1 0.0 ms 0% 0.0 ms 0%
FacetComponent 1 0.0 ms 0% 0.0 ms 0%
I'm looking for how to improve the speed of the search query. The
QueryElevation Component is taking too much time which is
unacceptable. The size of elevation file is only 1 Mb. I wonder other
people using this component without problems (related to speed)? Am I
using it the wrong way or there is a limit when using this component?
This is a huge number of docs you want to elevate - I'm sure this wasn't designed for that :-)!
Isn't there a common attribute of your docs e.g. salesrank, where you can then simply use the dismax handler to boost the docs? e.g. via bf=recip(sqrt(sales_rank_i),1,1,1))
(Or you could simply sort the docs against this salesrank field)
Or couldn't you introduce such a field?
Related
we have run 3000 samples total duration time is 4 min 45 sec how is it possible 99% of requests are taking time only 1 min 20 sec and only 1 % of requests take the rest of the time
we have run 3000 samples total duration time is 4 min 45 sec how is it possible 99% of requests are taking time only 1 min 20 sec and only 1 % of requests take the rest of the time
As per JMeter Glossary:
90% Line (90th Percentile) is the value below which 90% of the samples fall. The remaining samples too at least as long as the value. This is a standard statistical measure. See, for example: Percentile entry at Wikipedia.
As per Request Stats Report
90% line (ms) 90th Percentile. 90% of the samples were smaller than or equal to this time.
95% line (ms) 95th Percentile. 95% of the samples were smaller than or equal to this time.
99% line (ms) 99th Percentile. 99% of the samples were smaller than or equal to this time.
So if you see i.e. 1500 ms as 90% percentile it means that 90% of Samplers response time was 1500 ms or less, the remaining ones are higher.
I am running elastic search in my personal box.
Memory: 6GB
Processor: Intel® Core™ i3-3120M CPU # 2.50GHz × 4
OS: Ubuntu 12.04 - 64-bit
ElasticSearch Settings: Only running locally
Version : 1.2.2
ES_MIN_MEM=3g
ES_MAX_MEM=3g
threadpool.bulk.queue_size: 3000
indices.fielddata.cache.size: 25%
http.compression: true
bootstrap.mlockall: true
script.disable_dynamic: true
cluster.name: elasticsearch
index size: 252MB
Scenario:
I am trying to test the performance of my bulk queries/aggregations. The test case is to run asynchronous http requests to node.js which in turn will call elastic search. The tests are running from a Java method. Started with 50 requests at a time. Each request is divided and parallized in to two asynchronous(async.parallel) bulk queries in node.js. I am using node-elasticsearch api (uses elasticsearch 1.3 api). The two bulk queries contain 13 and 10 queries respectively.And the two are asynchronously sent to elastic search from node.js. When the Elastic Search returns, the query results are combined and sent back to the test case.
Observations:
I see that all the cpu cores are utilized 100%. Memory is utilized around 90%. The response time for all 50 requests combined is 30 seconds. If I run just the single queries each alone, in the bulk queries, each are returning in less than 100 milli-seconds. Node.js is taking negligible time to forward requests to elastic search and combine responses from elastic search.
Even if run the test case synchronously from java, the response time does not change. I may say that elastic search is not doing parallel processing. Is this because I am CPU or memory bound? One more observation: if I change heap size for elastic search from 1 - 3GB, the response time does not change.
Also I am pasting top command output:
top - 18:04:12 up 4:29, 5 users, load average: 5.93, 5.16, 4.15
Tasks: 224 total, 3 running, 221 sleeping, 0 stopped, 0 zombie
Cpu(s): 98.2%us, 1.0%sy, 0.0%ni, 0.8%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 5955796k total, 5801920k used, 153876k free, 1548k buffers
Swap: 6133756k total, 708336k used, 5425420k free, 460436k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
17410 root 20 0 7495m 3.3g 27m S 366 58.6 5:09.57 java
15356 rmadd 20 0 1015m 125m 3636 S 19 2.2 1:14.03 node
Questions:
Is this expected, because I am running Elastic Search in my local machine and not in a cluster? Can I improve my performance in my local machine? I would definitely start a cluster. But I want to know, how to improve the performance scalably. What is it that the elastic search is bound to?
I am not able to find this in forums. And am sure this would help others. Thanks for your help.
I've wanted to test StatsD perfromance with some stress test which I've made.
Eventually I found out that when there are something like 80,000 packets per sec the Node.js is getting to 100% cpu utilization on my environment.
I know that 80,000 event per sec is quite huge amount of events , but I wonder if anyone knows what are the limits of StatsD in regarding to Node.js. What is a normal events rate?
Also, is there something I can do to imporve Node.js performance so it won't get to 100% cpu utilization?
According to a StatsD contributer, the metrics rate which was tested and measured was between 15,000 - 20,000 metrics per sec.
Thats quite good enough and it is what I was looking for.
You can see more details about it in the issue I've opened at StatsD's github project:
https://github.com/etsy/statsd/issues/249
I'm wondering will my site perform loading faster if the images that I use to upload it on my wordpress wp-content-uploads now it's going to be store in a totally different domain (which I've control) that I installed the image upload services like flickr or any website that use for uploading service.
Because when I perform the test on http://gtmetrix.com I have like 93% for page speed and only 72% for Yslow
http://gtmetrix.com/reports/applesiam.com/rF12jFmv
I really think that my page load slow somehow because it says
Page load time: 14.54s
Total page size: 4.35MB
Total number of requests: 169
Thanks
Probably.
Most browsers throttles the number of simultaneous requests to each domain, so using multiple domains allows browsers to issue more requests at the same time. This should lead to a faster rendering of the page.
However, if obtaining an image from your domain is significantly slower than from the original one, than the change will not improve the load speed.
I don't think hosting your images on another hostname is going to make that much difference to your sites performance.
The site is slow because the pages are too large, and have many third party components - this waterfall will give you some idea -
http://www.webpagetest.org/result/120411_BG_b66024d1dda2429676d1e646e4b23a86/
Look at how you can reduce the number of components.
Updated:
Seriously look at the number of components and where they come from - out of 195 resources on the page only 31 come from applesiam.com
It will be faster. However the number of simultaneous connections from a browser to a domain is usually limited to 4-6.
You can circumvent this limit by creating a bunch of subdomain alias for your static content, all pointing to the same IP. Like
s[1-5].yourdomain.com -> same IP
Then access your images from different aldomains and the browser will load them parallel.
Update:
Looking at your site more thoroughly, responses from your site are slow.
Requesting http://applesiam.com/wp-content/themes/volt/images/social_sprite_32.png gives back HTTP 304 (Not modified) but that took 824 ms (tested with Firebug). It should be around 100 ms from the other side of the globe.
I made some ApacheBench tests for this image.
10 concurrent request:
» ab -n 10 -c 10 http://applesiam.com/wp-content/themes/volt/images/social_sprite_32.png
Percentage of the requests served within a certain time (ms)
50% 1683
66% 1685
75% 1701
80% 1710
90% 1710
95% 1710
98% 1710
99% 1710
100% 1710 (longest request)
20 concurrent request:
» ab -n 20 -c 20 http://applesiam.com/wp-content/themes/volt/images/social_sprite_32.png
Percentage of the requests served within a certain time (ms)
50% 2272
66% 2899
75% 2910
80% 2974
90% 2982
95% 2985
98% 2985
99% 2985
100% 2985 (longest request)
Doubling the number of simultaneous requests almost doubled the response time, which means your server cannot handle a moderate number of requests. You should check your server for bottlenecks (CPU, RAM, network, web sever config).
I am new to the Oracle 10g Resource Manager and am looking for guidance on how to put together a plan to meet my goals. I will test the plan, but I'm hoping for some guidance so I don't have to try hundreds of different plan configurations. I have the following goals:
Don't let non-sys sessions significantly slow down sys sessions.
Don't let any OLTP users sessions significantly slow down any other OLTP users sessions.
Don't let batch processing slow down OLTP.
Given these goals my first thought is to create the following consumer groups/plan directives:
Consumer Group Level 1 Level 2 Level 3
SYS 100% 0% 0%
OLTP1 0% 10% 0%
OLTP2 0% 10% 0%
OLTP3 0% 10% 0%
OLTP4 0% 10% 0%
OLTP5 0% 10% 0%
OLTP6 0% 10% 0%
OLTP7 0% 10% 0%
OLTP8 0% 10% 0%
OLTP9 0% 10% 0%
OLTP10 0% 10% 0%
BATCH 0% 0% 100%
Using this method each OLTP user could be put in a different OLTP group (assuming 10 users). The documentation isn't very clear on this, but it sounds like if an OLTP user in group OLTP1 needs more than it's 10% share that it will get it as long as every other OLTP group is getting 10% if it needs it. Is my understanding accurate? Will this work or is there a better way?
I would simplify this a little, make one group for OLTP, if they have the same requirements. Only make a new group when that new group has different requirements than the others in terms of priority. Also make sure that when an OLTP user has started a long running heavy duty process, that this session is switched to the batch group, or not started at all.
Resource manager only kicks in when cpu consumption is at 100%. From that point on it will start dividing resources to make sure that each group gets what it should get based on your directives.
Other things to think of are max parallel degree, session pool and (from 11g and up) undo usage and io limits.
best regards,
Ronald
http://ronr.blogspot.com