My application which was built in asp.net mvc 2 crashes sometimes (maybe once a month), and the error is Service Unavailable 503. Both times restarting app pool made the application work again. Since this error only happens live on the server (shared hosting) I don't know how to debug it. I don't have access to event logs so I don't see a way for debugging it.
Any suggestion?
UPDATE:
I contacted my hosting provider and they sent me this:
Imposed memory limit in Windows servers
Q: Is there a CPU/memory limit/restriction for Windows
Plans? A: Yes. 100MB for Reseller Class, 250MB for Personal Class ASP
and 500MB for Business Class ASP.
Q: What will happen if I hit the memory limit? A: If the worker
process exceeds the private memory quota, IIS will recycle that pool
which limits the memory usage. Your active sessions to the website may
get expired. If your site works on authentication you will be asked to
login again.
Q: How can I check the memory limit for my site? A: Run your site in
your local or test machine with the limits and try to optimize the
codes. If the limit of 250MB is exceeded you should get Business
Class, else you can go for Personal Class.
Q: What if my site exceeds the 500MB limit in Business Class? A: We
can increase the worker process to 2 if you are in a Business Class
server. If the number of worker processes are increased, the load will
be evenly shared across both the WP processes.
If the memory usage is still high, you should consider getting a
dedicated server where you can use unmetered memory for your website.
My account is Personal Class ASP (250MB). Since my website is a photo gallery, could this be related with generating thumbnails?
Thanks,
Ilija
Are you generating thumbnails on the fly when the request for thumbnail comes to the server? If yes, then you might want to consider generating thumbnails when you upload the photo and then just serve smaller images.
Very nice pictures, btw ;-)
PS. Try setting up memory limit on your local machine and hammer it with multiple requests - maybe you will be able to reproduce it.
Related
We have a memory problem with our Azure Windows App Service Plan (service level is P1v3 with 1 instance – this means 8 GB memory).
We are running two small .NET 6 App Services on it (some web APIs), that use custom containers – without problems.
They’re not in production and receive a very low number of requests.
However, when looking at the service plan’s memory usage in Diagnose and Solve Problems / Memory Analysis, we see an unexplained 80% memory percent usage – in a stable way:
And the real problem occurs when we try to start a third app service on the plan. We get this "out of memory" error in our log stream :
ERROR - Site: app-name-dev - Unable to start container.
Error message: Docker API responded with status code=InternalServerError,
response={"message":"hcsshim::CreateComputeSystem xxxx:
The paging file is too small for this operation to complete."}
So it looks like docker doesn’t have enough mem to start the container. Maybe because of the 80% mem usage ?
But our apps actually have very low memory needs. When running them locally on dev machines, we see about 50-150M memory usage (when no requests occur).
In Azure, the private bytes graph in “availability and performance” shows very moderate consumption for the biggest app of the two:
Unfortunately, the “Memory drill down” is unavailable:
(needless to say, waiting hours doesn’t change the message…)
Even more strange, stopping all App Services of the App Service Plan still show a Memory Percentage of 60% in the Plan.
Obviously some memory is being retained by something...
So the questions are:
Is it normal to have 60% memory percentage in an App Service Plan with no App Services running ?
If not, could this be due to a memory leak in our app ? But app services are ran in supposedly isolated containers, so I'm not sure this is possible. Any other explanation is of course welcome :-)
Why can’t we access the memory drill down ?
Any tips on the best way to fit "small" Docker containers with low memory usage in Azure App Service ? (or maybe in another Azure resource type...). It's a bit frustrating to be able to use ony 3GB out of a 8GB machine...
Further details:
First app is a .NET 6 based app, with its docker image based on aspnet:6.0-nanoserver-ltsc2022
Second app is also a .NET 6 based app, but has some windows DLL dependencies, and therefore is based on aspnet:6.0-windowsservercore-ltsc2022
Thanks in advance!
EDIT:
I added more details and changed the questions a bit since I was able to stop all app services tonight.
I have two .NET 4.6.1 MVC web applications, with identical code, that are running on IIS 10.0.14393.0 on Windows Server 2016. When viewing the worker process in IIS, the memory is around 150mb for Site A but 350mb for Site B. I know the code is identical because I copied the directory for Site A to create Site B. I've checked the application pools settings for both and they're identical with 0 for both Private Memory Limit and Virtual Memory Limit as well as all other settings being identical.
This is a dev server so no one else is accessing these applications. The only actions that were performed were displaying the login page. The application was not displayed in the Worker Process section until I went to the login page. Has anyone ever experienced this type of memory discrepancy before for 2 identical sites and have any suggestions for things to review?
I've also moved the code to a different server and its memory usage is similar to Site B. I've followed the instructions at the following site to look for memory leaks but it didn't find any issues. Which would make sense because this is not necessarily a memory leak but a site using a large amount of memory initially in one instance and not in another.
https://learn.microsoft.com/en-us/iis/troubleshoot/performance-issues/troubleshooting-native-memory-leak-in-an-iis-7x-application-pool
Based upon it allocating memory vastly differently between two applications running identical code with identical requests on the same server, I would suspect this is a server setting or IIS setting somewhere that I'm missing or just a base misunderstanding of how IIS allocates memory for the worker process. Any assistance or related resources would be greatly appreciated.
I also reviewed this article but my scenario is different as the identical sites are on the same server: IIS worker process making high memory usage
The issue was indeed related to a difference between Application Pool settings. In my initial comparison between Site A and Site B, I missed that Site A had Enable 32 Bit Applications set to True and Site B had Enable 32 Bit Applications set to False.
The lesson here is to double check after comparing settings. :-)
I have a site hosted on Windows Azure shared websites. It just got suspended for going over memory usage limit of 512MB/hour.
I do use .net caching rather heavily (to prevent multiple calls to database/external APIs, etc...).
Is that caching a no-no in shared websites on Windows Azure?
Do you use System.Runtime.Cache? You should be able to limit the amount of caching e.g. the memorycache object uses. See http://msdn.microsoft.com/en-us/library/dd941874.aspx for more information.
Even if you will stop using Cache it still can be used by framework/libs. I also have same problem (interesting, that in free mode memory limit is 1024MB, but shared one is lowered to 512).
As I see, memory amount that Azure shows on portal seems very close to System.Diagnostics.Process.GetCurrentProcess().PrivateMemorySize value.
At this moment I'm experimenting with caching settings to set maximum memory:
<system.web>
<caching>
<cache privateBytesLimit="250000000" privateBytesPollTime="00:00:15"/>
</caching>
</system.web>
Several days ago I set 300MB but several minutes ago got suspended again :(, so lowering to 250MB.
But anyway, this is very unclear, strange and "wrong" solution imho.
UPDATE
Got suspended again this morning. Temporarily converted to standard mode with small instance (1.7 GB RAM).
My WorkingSet counter now is about 200 megs now (with PeakWorkingSet 330 megs). BUT! GC's CollectionCount is increased approx 8 times (Gen0 is 1800 times instead of 250 for less that a day).
My current theory is that in "shared" mode websites are running inside "big" VM with a lot of memory and Garbage Collector just not have a need to run often, leading to longer "garbage life" and more memory consumption.
Have no access to my developer computer right now for some verification, but planing to convert site to web role in cloud service ASAP - with extra small instance (cost is comparable to shared web site cost)...
Might be worth checking a profile using perfmon on your local machine to see if what if its hitting the limits normally first, then look at maybe configuring the logging on Azure and again digging through it.
Also ensuring everything is precompiled and that your not loading and modules etc you don't need can really effect performance etc on Azure.
I think what you might want to try here is scale our instead of up. If you add a second instance that will double your resource limit.
Is there a max memory size for w3wp.exe? Mine is getting up to about 2.5-3G then seems to crash/reset.
Per the "GIVEN" dimensions below I setup some counters and noticed that the w3wp.exe will service http requests then reset to 0 along with the w3wp.exe process crashing (changing pids). As a result REQUESTS_QUEUED and ACTIVE_REQUESTS grow large causing delays in processing until the w3wp.exe can restart itself. It's doing this every 3-4min so more than likely due to heavy system volume during peak load. But not sure if it's a memory issue or not.
I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
RESULT: Customers are reporting sporadic response times for http requests.
Can I increase this memory limit or reconfigure IIS to handle increased load?
GIVEN:
System has been passed down to me so there may be gaps with IIS configuration, etc.
Database: SQL Server 2008R2
Web Servers: Windows Server 2008R2 Enterprise SP1 (64bit, 64G RAM)
IIS 7.5
Using MVC4 Web API with MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
Looked at the logs but really don't see anything significantly relevant
One application pool...no other LOB applications running on this server
Is the application pool set to run in 32-bit mode? That can cause memory issues even if you have plenty of RAM. On a 64-bit system, the memory limit for a 32-bit process is 4 GB.
Actually after solving the RC in which overuse of memorycacheing that was crashing the w3wpe.exe process I can safely say that an mvc4 web api service can grow up to 20G ... from baseline of 3G (64bit machine and application pool). AT least that was the last level I saw it before eviction policy starting cleaning up things. Probably a bit excessive in footprint but the application is very fast returning machine learning targeted content sub-100ms.
I have an asp.net mvc4 web api interface that gets about 54k requests a day.
http://myserv.x.com/api/123/getstuff?whatstuff=thisstuff
I have 3 web servers behind a load balancer that are setup to handle the http requests.
On average response times are ~300ms. However, lately something has gone awry (or maybe it has always been there) as there is sporadic behavior of response times coming back in 10-20sec. This would be for the same request hitting the same server directly instead of through the load balancer.
GIVEN:
- System has been passed down to me so there may be gaps with IIS confiuration, etc,.
- Database: SQL Server 2008R2
- Web Servers: Windows Server 2008R2 Enterprise SP1
- IIS 7.5
- Using MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
- Looked at the logs but really don't see anything significantly relevant
- One application pool...no other LOB applications running on this server
Assumptions & Ask:
Somehow I'm thinking that something is recycling the application pool or IIS worker threads are shutting down and restarting thus causing each new request to warmup and recache itself. It's so sporadic that it's tough to trouble shoot right now. The same request to the same server comes back fast as expected (back to back N requests) since it was cached in about 300ms....but wait about 5-10-20min and that same request to the same server takes 16seconds.
I have limited tracing to go by as these are prod systems so I can only expose so much logging details. Any help and information attacking this or similar behavior somebody else has run into is appreciated. Thx
UPDATE:
The w3wpe.exe process grows to ~3G. Somehow it gets wiped out and the PID changes so itself or something is killing it every 3-4min I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
After 4-5 days of assessing IIS and configuration vs internal code issues I finally found the issue with little to no help with windbg or debugdiag IIS tools. Those tools contain so much information even with mini dumps or log trace stacks that they can be red herrings. Best bet was to reproduce it by setting up a "copy intelligently" instance of a production system, which we did not have at the time and took a bit for ops to set something up.
Needless to say the problem had to do with over cacheing business objects. There was one race condition where updates on a certain table were updating an attribute to that corresponding business object (updates were coming from multiple servers) which was causing an OOC stackoverflow that pretty much caused the cacheing to recursively cache itself to death thus causing the w3wp.exe process to die and psuedo-recycle itself. It was one of those edge cases that was incredibly hard to test and repro in a non-production environment.