I'm in development for a Rails app and I am confused about what I'm seeing. I am new to this so I may misinterpreting the information. When I run one web process I am getting good results. But when I up the web process I am not getting the results I expect. I am trying to calculate how many I will need to run in production so I can determine my costs.
Based on New Relic I have response times of 40-60 MS per request at 3000 requests per minute (or about 50 requests per second) on one Heroku Dyno. Things are working just fine and the web processes are not even being pushed. Some responses are timing out at 1 second on Blitz, but I expect because I'm pushing as much as I can through one Dyno.
Now I try cranking up the Dyno's. First to 10 then to 50. I rush with Blitz again and get the same results as above. With 50 dynos running I blitz the website with 250 concurrent users and I get response times of 2 or 3 seconds. New Relic is reading the same traffic as one dyno, 3000 requests per second with 60MS request times. Blitz is seeing 3 second response times and getting a max of 50 to 60 rps.
In the logs I can see the activity split nicely between the different web processes. I am just testing against the home page and not accessing any external services or database calls. I'm not using caching.
I don't understand why a single web process easily handle up to 60 requests per second (per Blitz), but when increasing to 10 or even 50 web processes I don't get any additional performance. I did the Blitz rushes several times ramping up concurrent users.
Anyone have any ideas about what's happening?
I ended up removing the app server, Unicorn, from my app and adding it back in again. Now I'm getting 225 hits per sec with 250 concurrent users on one dyno. Running multiple Dyno's seems to work just fine as well. Must have been an error in setting up the App Server. Never did track down the exact error though.
Related
Getting 503 Error while Running the JMeter for the Thread User 400,Is it Because of Server issues.? When I run the thread group for 100 user with ramp up period 25 seconds then it will be working fine but for the user 400 users its giving 503 error.
Given you don't experience any issues with 100 users and have issues with 400 users most probably it's a server issue connected with the overload so congratulations on finding the bottleneck.
You can either report it as is or perform a little bit deeper investigation in order to find the cause, suggested steps:
Instead of kicking off 400 users at once try increasing the load gradually at the same time looking at Response Times vs Threads and Transaction Throughput vs Threads charts. Ideally response time should remain the same and throughput should be growing as the number of threads increase. When response time starts increasing and throughput starts decreasing it indicates the saturation point and at this stage you can state that this is the maximum number of users your application can support
Check your application logs and configuration as it might be not properly tuned for the high loads, you can use 15 Simple ASP.NET Performance Tuning Tips as a reference or look for a similar guide for your application technology stack
Ensure that your application has enough headroom to operate in terms of CPU, RAM, Network, etc. as it might be the case that it's basically a lack of resources, it can be done using i.e. JMeter PerfMon Plugin
Repeat your test with profiler tool telemetry in place, this way you will be able to localize the problem and state where is the problematic piece of code or inefficient algo lives.
If server isn't down/restarted, then yes, 503 indicate overload
Common causes are a server that is down for maintenance or that is overloaded
You need to find what stop server from serving 400 concurrent requests/users
Notice that if you are testing on a test environment which isn't equal/similar to production environment, it may not reflect the load that production server can endure
I have been testing some viewport issues for mobile and probably ran
git push heroku master
about 50 times in the last 3 hours. I am now seeing from the google speed tests that:
Reduce server response time In our test, your server responded in 8.9
seconds. There are many factors that can slow down your server
response time. Please read our recommendations to learn how you can
monitor and measure where your server is spending the most time. Hide
details
This wasn't popping up earlier this morning and was under .5 seconds. Did I damage one of my dynos on the heroku servers?= My site isn't really getting any traffic yet so I haven't been doing any stage testing.
What is the best way to test production?
I was reading through this but was wondering if there is a better way to test production quickly.
https://devcenter.heroku.com/articles/multiple-environments
Thanks,
Jeff
There's nothing wrong with pushing many times in a row, but every time you push, your dynos will cycle. This takes something like 5 to 15 seconds depending on the size of your slug.
Generally this means that the first query sent to your app at the moment your dynos are cycling might hang for about that long. If Google checked your server's speed at that time, then that explains the response time. However, there shouldn't be any lasting effects after you finish pushing repeatedly.
If I recall correctly there is a Heroku labs option to cycle dynos to eliminate this pause, basically taking down some of your dynos and then cycling them while the other ones are still up, but I do not recommend using it as it makes code pushes very unpredictable and can result in two versions of your app being live at the same time.
I have ASP.NET MVC app which accept file uploads and has result pooling using SignalR. The app hosted on Prod server with IIS7, 4 Gb Ram and two cores CPU.
The app on Dev server works perfectly but when I host it on Prod server with about 50 000 users per day the app become unrresponsible after five minutes of running. The web page request time increase dramatically and it takes about 30 seconds to load one page. I have tried to record all MvcApplication.Application_BeginRequest event call and got 9000 hits in 5 minutes. Not sure is this acceptable number of hits or not for app like this.
I have used ANTS Performance Profiler(not useful in Prod app profiling, slow and eats all memory) to profile code but profiler do not show any time delay issues in my code/MSSQL queries.
Also I have tried to monitored CPU and RAM spike problems but I didn't find any. CPU percentage sometimes goes to 15% but never up and memory usage is normal.
I suspect that there is something wrong with request or threads limits in ASP.NET/IIS7 but don't know how to profile it.
Could someone suggest any profiling solutions which could help in this situation? Tried to hunt the problem for two week already without any result :(
You may try using the MiniProfiler and more specifically the MiniProfiler.MVC3 NuGet package which is specifically created for ASP.NET MVC applications. It will show you all kind of useful information such as the time spend for different methods in the execution of the request.
I want to host a static website on Heroku, but not sure how many dynos to start off with.
It says on this page: https://devcenter.heroku.com/articles/dyno-requests that the number of requests a dyno can serve, depends on the language and framework used. But I have also read somewhere that 1 dyno only handles one request at a time.
A little confused here, should 1 web dyno be enough to host a static website with very small traffic (<1000 views/month, <10/hour)? And how would you go about estimating additional use of dynos as traffic starts to increase?
Hope I worded my question correctly. Would really appreciate your input, thanks in advance!
A little miffed since I had a perfectly valid answer deleted but here's another attempt.
Heroku dynos are single threaded, so they are capable of dealing with a single request at a time. If you had a dynamic page (php, ruby etc) then you would look at how long a page takes to respond at the server, say it took 250ms to respond then a single dyno could deal with 4 requests a second. Adding more dynos increases concurrency NOT performance. So if you had 2 dynos, in this scenario you be able to deal with 8 requests per second.
Since you're only talking static pages, their response time should be much faster than this. Your best way to identify if you need more is to look at your heroku log output and see if you have sustained levels of the 'queue' value; this means that the dynos are unable to keep up and requests are being queued for processing.
Since most HTTP 1.1 clients will create two TCP connections to the webserver when requesting resources, I have a hunch you'll see better performance on single clients if you start two dynos, so the client's pipelined resources requests can be handled pipelined as well.
You'll have to decide if it is worth the extra money for the (potentially slight) improved performance of a single client.
If you ever anticipate having multiple clients requesting information at once, then you'll probably want more than two dynos, just to make sure at least one is readily available for additional clients.
In this situation, if you stay with one dyno. The first one is free, the second one puts you over the monthly minimum and starts to trigger costs.
But, you should also realize with one dyno on Heroku, the app will go to sleep if it hasn't been accessed recently (I think this is around 30 minutes). In that case, it can take 5-10 seconds to wake up again and can give your users a very slow initial experience.
There are web services that will ping your site, testing for it's response and keeping it awake. http://www.wekkars.com/ for example.
IIS includes a worker process health check "ping" function that pings worker processes every 90 seconds by default and recycles them if they don't respond. I have an application that is chronically putting app pools into a bad state and I'm curious if there is any reason not to lower this time to force IIS to recycle a failed worker process quicker. Searching the web all I can find is people that are increasing the time to allow for debugging. It seems like 90 seconds is far to high for a web application, but perhaps I'm missing something.
Well the obvious answer is that in some situations requests would take longer than 90 seconds for the worker process to return. If you can't imagine a situation where this would be appropriate, then feel free to lower it.
I wouldn't recommend going too much lower than 30 seconds. I can see situations where you get in recycle loops. However you can do testing and see what makes sense in your situation. I would recommend Siege for load testing to see how your application behaves.