I have a .NET 1.1 application hosted on two different servers, but on one of them whenever the application pool is recycled, all sessions are dropped.
Both applications are using “StateServer” session mode and as far as I could tell, both servers have exactly the same configuration and have the “ASP .NET State Server” service running.
This is a particularly troublesome issue, due to the fact that this application pool is recycling every 2-3 hours (that’s another issue that I have to solve).
Does anyone have any idea of might be causing this?
Thanks in advance,
Zeon
Monitor the number of active user sessions in each State Server instance with the perfom counter "State Server Sessions Active" to (a) ensure that both servers are using STate Server and (b) see that this really is caused by the app pool recycling.
It's a bit umclear from your question if these two apps share session state, or if they are two different applications, that could be important for the solution.
Related
I have 2 web servers (IIS 8.5) behind a hardware firewall and our application uses SignalR for some real-time updates. We are using SQL Server as the backplane to help us work in this load balanced environment. Additionally we are using sticky sessions on the load balancer to help us keep the users on the same web server during their session. When we are running in this hardware configuration we lose at least 1/3 of our messages. Sometimes we get all the expected messages but more often than not we are missing plenty.
When we are running on a single web server all messages are received. Does anyone have any suggestions for troubleshooting this problem? We've turned on logs (both client & server) and nothing looks like it's missing or broken. We're really stumped.
EDIT---
Some additional details that I hope will shed light on the situation.
Server to Client messages are getting lost. Pretty much all our communication is Server to Client.
We are using sticky session just based on IP and limited to 5 minutes but we're losing messages within that 5 minutes.
This is some old SignalR code that has been only minimally touched since SignalR 1 (or even older). We are keeping an in memory list of users along with their connections and we use that list to send notices back to the client. It seems most likely that this is the cause of the troubles but with Sticky sessions the user should be stuck to the same server for at least the 5 minutes right?
This list of users maps Username to connection id. This is useful when our backend services (on another machine) sends a message back with the username not the connection id.
Finally resolved this. There were 2 issues really. The first is that we were using an in memory list of users as mentioned in the edit above. Once we realized that wasn't going to work across machines we removed it. It also led us to the second issue which was how SignalR 2 uses the IUserIdProvider and our call should have been
Clients.User(userId).send(message)
instead of
context.Clients.Client(connection)
This code had existed since we first started using SignalR many years ago and never got properly updated as we upgraded SignalR versions
Have the same machineKey specified in your web.config on both servers.
The issue I have is that I'm using a third party dll for something (very expensive operation), it's not serializable, and it takes a minute to spin up each time. It's needed on each call of a WCF service and I can't keep it in memory (recyling), and I can't keep it in a cache (unserializable).
I was wondering what alternatives (if any) there are? I was originally thinking about using a Worker Role, but then I read that they are recycled too. Then I considered a Windows service, but I'm hoping there is something better suited.
I'd like to think I'm not the only one with this issue, and that someone else has already solved this issue! :)
Why are you unable to use Worker Roles or Web Roles to keep the data generated by yoru process in memory? Neither of the two roles "flushes" it's memory on a frequent basis. True, that it is not guaranteed that reboots do not happen, but those reboots are very rare and checking to see IF your statefull data is empty and then repopulating it when it is, shouldnt be a big deal and the logic would work on any server the same way, whether it is a Cloud Service or a dedicated VM.
Edit: Web roles or worker roles do not restart on any known cycle. However, by default IIS does recycle on a schedule. This timer can be changed or disabled via a startup script.
Furthermore, no such recycling happens in worker roles. So, if you're running a worker role, the thing will stay in memory as long as you dont recycle the server yourself or a rare windows update happens
HTH
I have an asp.net mvc4 web api interface that gets about 54k requests a day.
http://myserv.x.com/api/123/getstuff?whatstuff=thisstuff
I have 3 web servers behind a load balancer that are setup to handle the http requests.
On average response times are ~300ms. However, lately something has gone awry (or maybe it has always been there) as there is sporadic behavior of response times coming back in 10-20sec. This would be for the same request hitting the same server directly instead of through the load balancer.
GIVEN:
- System has been passed down to me so there may be gaps with IIS confiuration, etc,.
- Database: SQL Server 2008R2
- Web Servers: Windows Server 2008R2 Enterprise SP1
- IIS 7.5
- Using MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
- Looked at the logs but really don't see anything significantly relevant
- One application pool...no other LOB applications running on this server
Assumptions & Ask:
Somehow I'm thinking that something is recycling the application pool or IIS worker threads are shutting down and restarting thus causing each new request to warmup and recache itself. It's so sporadic that it's tough to trouble shoot right now. The same request to the same server comes back fast as expected (back to back N requests) since it was cached in about 300ms....but wait about 5-10-20min and that same request to the same server takes 16seconds.
I have limited tracing to go by as these are prod systems so I can only expose so much logging details. Any help and information attacking this or similar behavior somebody else has run into is appreciated. Thx
UPDATE:
The w3wpe.exe process grows to ~3G. Somehow it gets wiped out and the PID changes so itself or something is killing it every 3-4min I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
After 4-5 days of assessing IIS and configuration vs internal code issues I finally found the issue with little to no help with windbg or debugdiag IIS tools. Those tools contain so much information even with mini dumps or log trace stacks that they can be red herrings. Best bet was to reproduce it by setting up a "copy intelligently" instance of a production system, which we did not have at the time and took a bit for ops to set something up.
Needless to say the problem had to do with over cacheing business objects. There was one race condition where updates on a certain table were updating an attribute to that corresponding business object (updates were coming from multiple servers) which was causing an OOC stackoverflow that pretty much caused the cacheing to recursively cache itself to death thus causing the w3wp.exe process to die and psuedo-recycle itself. It was one of those edge cases that was incredibly hard to test and repro in a non-production environment.
When I check my Tomcat Application Manager, I see that I have too many sessions (over 4600):
Why is that so ? Can this be linked to a deployed application or is it the fact that it has been up for a long time ?
There can be multiple reasons for this, some more exotic than others. Some of them:
you really have a lot of users coming to your site
your app ran out of memory and the thread which invalidates sessions has died
you've been hit by a search engine crawler or DDOSed (not very likely)
you have some sort of monitoring enabled which is done via HTTP and the sessions are not discarded
Have you looked at the list of these sessions? Do they have similar idle times? What pages are being hit? What do the HTTP logs say? (Google) Analytics?
The fact that the app has been online for a long time has no correlation to the session count. Tomcat invalidates these sessions on a regular basis.
In order to get the most precise answer do a memory dump and analyze it using the profiler.
This is a bit of a throwback question, and probably relatively fundamental, but I'm at a loss.
How does IIS manage Classic ASP session state?
We have an app that stores user information in session, and when many users are using the app, it seems to be recycling session for users, even though the "expire period" has not elapsed.
We suspect that when a certain amount of memory has been used for the session state, it begins to recycle the oldest session objects or something like this.
If this is correct, is there some way to control for it with the existing application code?
Thanks!
ASP sessions are stored as simple in memory COM objects when the process hosting the ASP application are terminated so will all the sessions.
ASP does not "recycle" active sessions. However there are number of other circumstances which can affect ASP sessions.
Application Pool Idle Timeout
One phantom reason "Sessions" appear to timeout prematurely is because the "Sessions" in question are just under test during development. Hence whilst the developer is examining the content of a page or reviewing some code no further requests hit the site since its not actually a live site.
In IIS manager open the properties of the pool in which your ASP application runs. Take a look at the Performance tab. The Idle Timeout will default to 20 minutes. Hence if you have specified a session timeout of say 60 minutes and you are "testing" that timeout you actually discover your session has timed-out in 20 minutes. The lack of activity has killed the application pool.
Application Pool Recycling
IIS may recycle the application pool in which the ASP application is running in. Recycling means that the existing set of processes currently hosting the ASP application no longer accept new requests. New requests go to a new set of processes and the older processes will terminate when they have completed their outstanding requests.
There are a whole host of different settings and criteria that can be configured that trigger the recycling of an application pool. Take a look at the Recycling tab of the pool properties dialog.
If you think that there may be an excessive demand for memory then the Memory recycling section may indicate a cause.
Web Garden
An Application Pool can contain multiple processes to run the same set of applications. Back on the performance tab note the Web Garden section at the bottom. By default this is set to 1. However multiple worker processes will play havoc with ASP sessions. As noted above ASP session are simple in-memory COM objects. If subsequent requests for a specific session are dished out to different workers one worker will not have access to the session object that the other has.
Session.Abandon or Session.Clear
Logic bugs can sometimes be the cause of sessions apparently disappearing. Calling the above methods at an inappropriate point in a sessions life can cause a problem.
I have experienced the same thing. Session seems to be emptied of the data, meaning that no variables is no longer stored in the session, but since the session exists, On_SessionStart doesn't trigger.
Gives you a headache if you initialize data for a visitor that you later on depends on...
I have considered this a bug that no one seems to know about, and haven't found a solution to it. It seems related to memory-usage, as you point out, and the solution seems to be to make sure you don't have any leaks.
Implement object-caching in classic ASP memory-leaking
This issue for me turned out to be the number of worker processes under the Performance tab. It was set to 2 for some reason. We set it back to 1 and the issue went away.