After searching to no end, as well as countless hours of trial and error with different settings, I've come up completely empty on why my server is performing so slowly.
Here are the basics. I've switched hosting from a local server (CF8 running on ubuntu) to a better equipped hosting company (CF10 running on Windows Server 2008). Both servers ran Xeon processors. My old linux server ran on 8GB ram. Windows is running on 9GB. Both are running 64-bit. The problem I am having is on a very simple task: initial CFC creation.
I have a custom created CMS, that runs 2 sets of CFC (application and session scoped) for the general public, only application scope CFC's are created, and when a user logs into the site, additional session scoped CFC's are created (anywhere from 8 - 16 depending on the number of modules the site contains).
On the linux box this worked great, fast with no issues. However, since switching to the Windows server and CF10, the creation process has become dreadful. When I go to log into a site, authentication is done, and the CFC's are created. When I first log into a site, this process can take anywhere from 15 - 50 seconds. When I log out, the session scope variables are all killed. If I was to log in a 2nd time, within a short period of time, my login time runs about 1 - 5 seconds depending on server load.
My initial thinking is that it's a memory allocation issue, but I'm running out of ideas. I have some of the following specs
JVM - 1.7.0_40
JVM Heap Size 1280
PermSize 256m
Simultaneous request limit 100
CFC request limit 60
cfthread pool size 50
trusted cache is currently off
I've set worker threads in IIS to have 5 per application. Each worker process runs at about 12,000k.
If anyone could help, it would be greatly appreciated.
Related
What happens is that a web service on my IIS server significantly increases the ram used.
It works in the range of 200 ~ 700 mb. But for a few days now, he suddenly starts using 3, 4, 5 gb of ram.
As a palliative solution to not block users, I end the service by the task manager itself and it goes back to normal, but some time later it increases again:
task manager photo
I used the performance monitor and saw that it increases this part here:
performance monitor photo
I really don't know how to solve this, I'm stuck, can anyone help me?
There are a lot of reasons that your IIS worker process could be using a lot of CPU, to start, you should look at which web requests are currently executing with IIS to see if that helps you identify the issue to be able to troubleshoot IIS worker process.
Via the IIS Worker Processes
Via the IIS management console, you can view the running worker processes. You can view which IIS application pool is causing high CPU and view the currently running web requests. after selecting "Worker Processes" from the main IIS menu, you can see the currently running IIS worker processes. If you double-click on a worker process, you can see all of the currently executing requests.
Scenario taking into account login user->Navigate to Page 01->hold the user for 5min->Logout user
Scripted as below:
Navigate to the Home page
The user is logged in (Assertion for login verification over some text on the dashboard)
Dashboard appears
Navigate to Page 01 (Assertion Page 01 content)
Logout (Constant Timer added for 5min and Assertion for logout to verify home page is redirected)
Step Up thread configuration has been kept this way:
For achieving this scenario distributed system was implemented as follows:
Master(My own machine 8 GB Ram and Core 2 Duo Processor)
2 slaves machines (8 GB Ram each and I7 and Core 2 Duo Processors)
Thread: jp#gc - Stepping Thread Group
The server has been configured as below:
2 EC2 Instance (16 GB Ram each)
1 Load Balancer
1 RDS Instance
Note: Instance is auto scaled at 60% CPU Utilization.
While executing the script for 500 concurrent users using stepping thread on Non-GUI mode, below list of error is appearing on the dashboard report
504/Gateway Time-out
Non HTTP response code: java.net.SocketException/Non HTTP response message: Connection reset
Assertion as logout failed
Could someone help me out to know why these are appearing? when I checked the Load Balancer 504/Gateway Time-out was not appearing there? I was trying to track these error but was not able to figure it out why these along with other two errors are appearing. When the same script is executed for 10 users not error is appearing on GUI mode.
While the same script when executed for 100-250 concurrent user it works pretty well when no such above error.
If the issue doesn't happen for 250 virtual users and happens for 500 - it's definitely the bottleneck caused by increased load, you just need to find out the reason.
Make sure to have DNS Cache Manager added to your Test Plan otherwise you may run into the situation when the load goes to one server only
Set up monitoring of your EC2 instances to ensure that they have enough headroom to operate in terms of CPU, RAM, Network, etc. You can use Amazon CloudWatch or JMeter PerfMon Plugin for this.
You might want to re-run the test with profiling tool telemetry enabled - this way you will be able to see where application spends the majority of time
Inspect the configuration of your application servers, databases, etc. as it might be configuration issue of the middleware
Be aware that according to JMeter Best Practices you should always be using the latest JMeter version so consider migrating to JMeter 5.0 (or whatever is the latest version available at JMeter Downloads page) on as soon as possible.
I have deployed a simple Spring boot app in Google App Engine Flexible. The app. has two APIs, one to add the user data into the DB (xxx.appspot.com/add) the other to get all the user data from the DB (xxx.appspot.com/all).
I wanted to see how GAE scales for the load, hence used JMeter to create a load with 100 user concurrency ramped up in 10 seconds and calls these two APIs in half a second delay, forever. While it runs fine for sometime (with just one instance), it starts to fail after 30 seconds or so with a "java.net.SocketException" or "The server responded with a status of 502".
After this error, when I try to access the same API from the browser, it displays,
Error: Server Error
The server encountered a temporary error and could not complete your
request. Please try again in 30 seconds.
The service is back to normal after 30 mins or so, and whenever the load test happens it repeats the same behavior as mentioned above. I expect GAE to auto-scale based on the load coming in to handle it without any down time (using multiple instances), instead it just crashes or blocks the service (without any information in the log). My app.yaml configuration is,
runtime: java
env: flex
service: hello-service
automatic_scaling:
min_num_instances: 1
max_num_instances: 10
I am a bit stuck with this one, Any help would be greatly appreciated. Thanks in advance.
The solution was to increase the resource configuration, details below.
Given that I did not set a resource parameter, it defaulted to the pre-defined values for both CPU and Memory. In this case, the default
memory was set at 0.6GB. App Engine Flex instances uses about 0.4GB
for overhead processes. Given Java is known to consume higher memory, there is a
great likelihood that the overhead processes consumed more than the
approximate 0.4GB value. Now instances in App Engine are restarted due
to a variety of reasons including optimization due to memory use. This
explains why your instances went off and it shows Tomcat is starting
up (they got restarted) and ends up in 502 error due to the nginx is
not able to complete the request. Fixing the above may lessen if not completely eliminate the 502s.
After I have specified the resources attribute and increased the configuration in app.yaml 502 error seems to be gone.
OS: Windows Server 2012 Standard
IIS: 8.0.9200.16384
Processor: 4x Xeon 2.67Ghz CPU
RAM: 40GB
Problem:
We have recently enabled IIS's AutoStart feature, since doing so our start up time for the application pools has gone up considerably. The application pool appears to be running but it seems to ramp up its CPU usage to the maximum 25% for about 30 minutes and the websites running in that pool don't respond until this has completed. We have checked the event log and there doesn't appear to be any faults. We have checked the logging in our preload function and this appears to only take about 60-90 seconds.
How can we diagnose what is causing the delay in the application pools starting up?
Background:
We are serving up multiple copies of the same ASP.Net MVC3 application, from multiple application pools (20 sites per pool). We have approximately 8 pools service up 160 sites We have IProcessHostPreloadClient built which preloads some settings from the database when the sites starts up. We have a second server with the same basic specs but only 3 pools of 20, which only takes approx 5 minutes per pool to start up.
For anyone interested here is what we did to resolve/mitigate the issue:
Break up our sites into smaller groupings per application pools (this reduces the startup time per application pool). We went with 10 sites per pool.
Change to using the IIS 8 Application Initialize 'PreloadEnabled' option rather than the serviceAutoStartProvider for site initialization.
When deploying new code, don't restart the application pools, instead use the app_offline.htm feature to unload the application and restart it.
The app_offline.htm feature is the key one for us, this means we are able to deploy new versions of our software with out stopping and starting the application pools and incurring the start up time penalty. Also incrementally restarting application pools help reduce the strain on the CPU which meant we got a consistent start up time for each pool. This is only required when we do an IIS reset or server restart (rarely).
Is there a max memory size for w3wp.exe? Mine is getting up to about 2.5-3G then seems to crash/reset.
Per the "GIVEN" dimensions below I setup some counters and noticed that the w3wp.exe will service http requests then reset to 0 along with the w3wp.exe process crashing (changing pids). As a result REQUESTS_QUEUED and ACTIVE_REQUESTS grow large causing delays in processing until the w3wp.exe can restart itself. It's doing this every 3-4min so more than likely due to heavy system volume during peak load. But not sure if it's a memory issue or not.
I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
RESULT: Customers are reporting sporadic response times for http requests.
Can I increase this memory limit or reconfigure IIS to handle increased load?
GIVEN:
System has been passed down to me so there may be gaps with IIS configuration, etc.
Database: SQL Server 2008R2
Web Servers: Windows Server 2008R2 Enterprise SP1 (64bit, 64G RAM)
IIS 7.5
Using MVC4 Web API with MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
Looked at the logs but really don't see anything significantly relevant
One application pool...no other LOB applications running on this server
Is the application pool set to run in 32-bit mode? That can cause memory issues even if you have plenty of RAM. On a 64-bit system, the memory limit for a 32-bit process is 4 GB.
Actually after solving the RC in which overuse of memorycacheing that was crashing the w3wpe.exe process I can safely say that an mvc4 web api service can grow up to 20G ... from baseline of 3G (64bit machine and application pool). AT least that was the last level I saw it before eviction policy starting cleaning up things. Probably a bit excessive in footprint but the application is very fast returning machine learning targeted content sub-100ms.