When I check my Tomcat Application Manager, I see that I have too many sessions (over 4600):
Why is that so ? Can this be linked to a deployed application or is it the fact that it has been up for a long time ?
There can be multiple reasons for this, some more exotic than others. Some of them:
you really have a lot of users coming to your site
your app ran out of memory and the thread which invalidates sessions has died
you've been hit by a search engine crawler or DDOSed (not very likely)
you have some sort of monitoring enabled which is done via HTTP and the sessions are not discarded
Have you looked at the list of these sessions? Do they have similar idle times? What pages are being hit? What do the HTTP logs say? (Google) Analytics?
The fact that the app has been online for a long time has no correlation to the session count. Tomcat invalidates these sessions on a regular basis.
In order to get the most precise answer do a memory dump and analyze it using the profiler.
Related
We are working on an ASP.NET 5 Web API project that is in production now but we are experiencing an issue where it becomes unresponsive intermittently throughout the day.
A few notes about the application architecture. It is an ASP.NET Web API project using a MariaDB database on a separate EC2 instance within the same private network. The connection string uses the private IP of the database server to avoid any name resolution issues. The site is hosted via IIS 10.
The application itself has been developed carefully following the best practices provided by Microsoft. Heavy focus on async operations, minimizing query response times and offloading more expensive operations into background services.
The app is extremely responsive. It performs with sub 100ms responses on almost all requests, even the more complicated requests, and all the way up until it becomes unresponsive this high level of performance remains the same. We tend to see between 10-30 requests per second and 300-500 select queries per second at peak usage so not too extreme. However, randomly (2-3 times over a 24 hour period) it will begin hanging on requests and simply not respond to the request. During this time, the database is still extremely responsive and we are never over 300 connections out of our 512 connection limit.
The resources on the application server itself are never really taxed much at all. The CPU never gets above ~20% and the memory usage sits around 20-30%.
If I were to stop the site in IIS and start it again while this is happening, it will quickly come back online. If I don't it will be down for a few minutes until IIS finally kills it due to a failed health check. There are no real errors generated as a response to the issue other than typical errors caused by the hanging of the process such as connection terminated errors. The only thing I have seen before that gave me pause was the fact that there a few connection timeouts when getting the connection from the pool, but like I said, the connections to the server are never close to the limit.
Also, this app and version has been in production for months and it wasn't until the traffic volume started to grow that we started seeing these issues. At this point, I am at a loss for next steps of troubleshooting and I'm seeking suggestions.
In IIS App Pool advanced settings set Start Mode to AlwaysRunning
I never found a root cause for this issue, however, after updating to newer versions of .NET MVC this issue went away. My best guess is that changes with the Kestrel possibly resolved this issue, although, I have no idea what specific change that might have been. I have gone through the change logs a few times and didn't see anything that specifically jumped out at me.
I'm looking for a way of capping number of sessions created for crawlers and/or anonymous users on my webapp (running on Jetty). I want to decrease number of sessions on the app. There are multiple sessions created for crawlers as many of them do not respect cookies. That would also help when app gets under DDoS.
There is already a solution for tomcat ( 'crawler session valve') but I need solution for Jetty.
What would be your recommendation on capping unwanted sessions?
Thanks,
JB
Use the Jetty DoS Filter:
"The Denial of Service (DoS) filter limits exposure to request flooding, whether malicious, or as a result of a misconfigured client. The DoS filter keeps track of the number of requests from a connection per second. If the requests exceed the limit, Jetty rejects, delays, or throttles the request, and sends a warning message. The filter works on the assumption that the attacker might be written in simple blocking style, so by suspending requests you are hopefully consuming the attacker's resources. The DoS filter is related to the QoS filter, using Continuations to prioritize requests and avoid thread starvation."
http://wiki.eclipse.org/Jetty/Reference/DoSFilter
Other configs that are worth looking at can be found in the Jetty Optimization Article:
http://67-23-9-112.static.slicehost.net/doc/optimization.html
We are experiencing slow processing of requests under heavy load. When looking at the currently running requests during these bursts I can see many requests to our web-service code.
The number of requests is not that large but they appear to be stuck in a preprocessing state. Below is an example:
We are running an IIS7 app pool in classic mode due to the need to support some legacy code.
Other requests continue to be processed but these stuck requests gradually seem to fill up the available threads leading to slow processing of other pages.
Does anyone have any idea on where these requests are getting stuck.
There appears to be no resource issue with the DB and the requests state show suggest this is all preprocessing.
We have run load tests on the code involved on local machines and can not replicate the issue.
Another possible factor is we are making use of MVC and UrlRouting.
Many thanks for any help.
Some issues only happen at production servers unfortunately, as load test can never simulate real world users.
You can try to capture hang dumps when performance is bad, and then analyze them (on your own or open a support case via http://support.microsoft.com to work with Microsoft support).
Usually you might have hit the famous thread pool bottleneck, http://support.microsoft.com/kb/821268. Dump analysis can easily tell the culprit and help locate a solution.
Why not move them into their own AppPool to separate them from the Classic ASP app - you'll then have more options to tune.
I am experiencing an issue with delayed writes to the application logs for a Java EE web application running in IBM WebSphere v. 7.x. Logging statements taking up to an hour to appear in the application logs.
The problem doesn't appear related to heavy loads; WAS is responding to page requests almost instantly, and I am testing against a box that isn't used for performance testing, and on a holiday no less -- there is very little activitiy on the server.
My guess would be that the thread associated with logging has been configured with very low priority, but I cant figure out where that would be configured via the admin console or the configuration files.
Has anyone else experienced this sort of issue with WebSphere?
it's possible you don't even enough available threads in the thread pool. Its consistant with the page requests being fast, as they are controlled by the WebContainer threads.
Try increasing it:
Servers > Application Servers > Thread pools > ...
Not sure exactly which one to increase its max value. In worst case, increase'em all. Increase it heavily, so to be sure.
Other options:
make sure you enough disk space / try to connect with jConsole to inquire.
This is a bit of a throwback question, and probably relatively fundamental, but I'm at a loss.
How does IIS manage Classic ASP session state?
We have an app that stores user information in session, and when many users are using the app, it seems to be recycling session for users, even though the "expire period" has not elapsed.
We suspect that when a certain amount of memory has been used for the session state, it begins to recycle the oldest session objects or something like this.
If this is correct, is there some way to control for it with the existing application code?
Thanks!
ASP sessions are stored as simple in memory COM objects when the process hosting the ASP application are terminated so will all the sessions.
ASP does not "recycle" active sessions. However there are number of other circumstances which can affect ASP sessions.
Application Pool Idle Timeout
One phantom reason "Sessions" appear to timeout prematurely is because the "Sessions" in question are just under test during development. Hence whilst the developer is examining the content of a page or reviewing some code no further requests hit the site since its not actually a live site.
In IIS manager open the properties of the pool in which your ASP application runs. Take a look at the Performance tab. The Idle Timeout will default to 20 minutes. Hence if you have specified a session timeout of say 60 minutes and you are "testing" that timeout you actually discover your session has timed-out in 20 minutes. The lack of activity has killed the application pool.
Application Pool Recycling
IIS may recycle the application pool in which the ASP application is running in. Recycling means that the existing set of processes currently hosting the ASP application no longer accept new requests. New requests go to a new set of processes and the older processes will terminate when they have completed their outstanding requests.
There are a whole host of different settings and criteria that can be configured that trigger the recycling of an application pool. Take a look at the Recycling tab of the pool properties dialog.
If you think that there may be an excessive demand for memory then the Memory recycling section may indicate a cause.
Web Garden
An Application Pool can contain multiple processes to run the same set of applications. Back on the performance tab note the Web Garden section at the bottom. By default this is set to 1. However multiple worker processes will play havoc with ASP sessions. As noted above ASP session are simple in-memory COM objects. If subsequent requests for a specific session are dished out to different workers one worker will not have access to the session object that the other has.
Session.Abandon or Session.Clear
Logic bugs can sometimes be the cause of sessions apparently disappearing. Calling the above methods at an inappropriate point in a sessions life can cause a problem.
I have experienced the same thing. Session seems to be emptied of the data, meaning that no variables is no longer stored in the session, but since the session exists, On_SessionStart doesn't trigger.
Gives you a headache if you initialize data for a visitor that you later on depends on...
I have considered this a bug that no one seems to know about, and haven't found a solution to it. It seems related to memory-usage, as you point out, and the solution seems to be to make sure you don't have any leaks.
Implement object-caching in classic ASP memory-leaking
This issue for me turned out to be the number of worker processes under the Performance tab. It was set to 2 for some reason. We set it back to 1 and the issue went away.