My sites get down every 2-3 days. It doesn't show any error on upfront, the browser keeps on loading for a very long time, but no data appears. When I check the apache error logs I found Max Request Workers limit exhausted. For the last 10 days, I am increasing the same the frequency is increased to 5days but still getting down. The site was launched 45 days ago, running perfectly for 30 days. Even we have not observed any hike in the traffic. The site is hosted at the AWS plan is t2.2xlarge.
Do you use many filters for layered navigation? When bots hit it if using sql search it will exceed max connections and lock things up and repeat over and over. One possible area to look at. I already had this issue and had to block all bad bots in robot.txt. Check mostly for Chinese bots and block by IP in htaccess or firewall tune robot.txt to instruct delay 10 for bots. Connect your site to cloudflare and tune things to disallow huge hits. In general, mostly Chinese bots are the ones who don't respect rules and robot.txt si personally blocked all China.
Related
We are working on an ASP.NET 5 Web API project that is in production now but we are experiencing an issue where it becomes unresponsive intermittently throughout the day.
A few notes about the application architecture. It is an ASP.NET Web API project using a MariaDB database on a separate EC2 instance within the same private network. The connection string uses the private IP of the database server to avoid any name resolution issues. The site is hosted via IIS 10.
The application itself has been developed carefully following the best practices provided by Microsoft. Heavy focus on async operations, minimizing query response times and offloading more expensive operations into background services.
The app is extremely responsive. It performs with sub 100ms responses on almost all requests, even the more complicated requests, and all the way up until it becomes unresponsive this high level of performance remains the same. We tend to see between 10-30 requests per second and 300-500 select queries per second at peak usage so not too extreme. However, randomly (2-3 times over a 24 hour period) it will begin hanging on requests and simply not respond to the request. During this time, the database is still extremely responsive and we are never over 300 connections out of our 512 connection limit.
The resources on the application server itself are never really taxed much at all. The CPU never gets above ~20% and the memory usage sits around 20-30%.
If I were to stop the site in IIS and start it again while this is happening, it will quickly come back online. If I don't it will be down for a few minutes until IIS finally kills it due to a failed health check. There are no real errors generated as a response to the issue other than typical errors caused by the hanging of the process such as connection terminated errors. The only thing I have seen before that gave me pause was the fact that there a few connection timeouts when getting the connection from the pool, but like I said, the connections to the server are never close to the limit.
Also, this app and version has been in production for months and it wasn't until the traffic volume started to grow that we started seeing these issues. At this point, I am at a loss for next steps of troubleshooting and I'm seeking suggestions.
In IIS App Pool advanced settings set Start Mode to AlwaysRunning
I never found a root cause for this issue, however, after updating to newer versions of .NET MVC this issue went away. My best guess is that changes with the Kestrel possibly resolved this issue, although, I have no idea what specific change that might have been. I have gone through the change logs a few times and didn't see anything that specifically jumped out at me.
I am using a shared hosting plan at Bluehost to host a golf tournament live scoring mobile web app. I am caching everything I can on Cloudflare, and spent quite some time on overall optimization of the initial download & rendering times. There might be more I could do, but without question my single biggest issue is the initial call to my website: www.spanishpointscup.org. Sometimes this seems to be related to DNS lookup and other times related to Waiting(TTFB).
Below are 2 screen shot images of the network calls that show variations in accessing my index.html. Sometimes this initial file load can be even longer. Very rarely are any of the other files downloaded creating a long delay time, so my only focus now is the initial file load. I think that even if I had server side rendering, I would still have this issue.
Does anyone have specific recommendations that they are confident will help me? Switch to VPS or other host? Thank you.
This is typical when you use a shared server.
The DNS has nothing to do with the issue. DNS has to do with the request not the response. It is the Browser that must resolve the the domain name to an ip address.
The delay you are seeing is due to the server being busy and your page is sitting in a queue waiting behind other processes. Possibly you have a CPU grabbing neighbor on your shared server. Or Bluehost has some performance issues.
You will likely notice some image files take an excessively long time to transmit. Which image is slow will appear to be random with each fresh (not in cache) page load.
UPDATE
After further review I noticed the "wait" times are excessive. Wait time is in green on your waterfall. Notice how the transmit time (blue) is short. This is the time it takes the server to retrieve the page from the disk and put it into the transmit buffer. 300-400 millisecond is excessive.
Find a new service provider.
I have a Windows Phone Google Tasks app that has been growing in popularity and a couple months ago I realized I was hitting the Google Tasks API courtesy quota of 5,000 daily requests.
I was getting 403 "DailyLimitExceeded". Following the advice of multiple forums, I requested an increase and was eventually granted 20,000 daily requests.. However, it still seems that the 5,000 limit is blocking me. To get around this I have created two more projects and have my app round robin users between them to help balance load.
I am using OAuth so I don't think I'm hitting the anonymous have limits. The projects all list the quota as 20,000 and track towards that but my original project starts erroring right around 5,000.. I have not hit that high of usage on the new projects yet. I have also tried turning off the API and back on, but that didn't help.
Any ideas on how to fix this?
There was a bug in the quota increase process that prevented changes from going into effect. The bug has been fixed in code and should be released early next week.
While our website is not yet complete graphically and design wise, most of the backend operations are near completion.
However, after optimising the mysql database we are still receiving a significant initial receiving period when tested on pingdom.com:
http://tools.pingdom.com/fpt/#!/IuoBna86v/http://foscam-uk.com
According to Pingdom:
The yellow part is the time it takes to resolve the hostname and similar (before the connection is initiated to the web server), the green part is connecting to the web server, and the blue part is the time it takes to retrieve the content from the webserver.
Upon asking our managed VPS support team we got the response : 'Have you tried optimizing your script? I believe that the high wait time on there indicates actual website loading time (meaning for your script to load); not actual connection to the website/server.'
Now, pingdom shows the js/css loading relatively quickly, the mysql database side of things doesn't seem to be slowing anything down either - does anyone have any suggestions of what this could be or might be causing it?
Thank you very much for your time and help.
89 requests are too many.
Reduce number of image request by creating sprites.This is pretty important from what is shown in pingdom.
Keep Alive should be set to On and Keep alive time should be a bit higher(15 seconds or so).
Use of compiler plus merge and minify js/css is recommended.
Change the hosting provider. 8 second loading is very very slow. It means that it actually is around 15-17 seconds for a user that doesn't have cached parts of your site (first time visitor). My site www.bebepunk.ro loads according to pingdom in 2.5 seconds and users still complain about the slowness of the site. Check also with http://www.webpagetest.org for both values.
We're running 3 different Drupal (Pressflow to be specific) sites on the same server. The 2nd and 3rd sites were cloned from the first one and load just fine. The first one, though, is taking a few seconds to connect and start sending data back from the server. Same box, same config (as far as we know), same modules, and generally the same theme. Here's what Pingdom shows...
Fast site:
http://i.stack.imgur.com/YZilC.png
Slow site:
http://i.stack.imgur.com/6Um1M.png
Edit: Those are from Pingdom, the yellow indicating "The web browser is waiting for data from the server"
The configs are the same, performance options same, server configs, as far as we can tell are the same. The delay occurs before any page elements are visible so it's not an on-page object problem or a page speed problem.
Could this be a config issue with the server? Where should we be investigating?
Thanks in advance!
I would try increasing the size of your MySQL cache. It's possible the fast sites have their particular queries cached and the slow site has a query that doesn't quite fit the cache, so the MySQL query results need to be regenerated each time.
Just a guess!