In a nutshell: my JBoss instance is running ok, but after some days it's performance is slowly degrading.
Detailed:
I've got a setup with JBoss 5.1.0-GA and Java 1.6.0_18-b07 (x64) running on a 64 bits RHEL 4 box. The hardware is a virtual machine with 8 core Xeon X5550 / 20G ram.
The product deployed in JBoss contains a webservice on which a endurance test is performed.
No database is involved in the process.
The tests are performed using soapui with 4 threads and the tests are configured to create 20% cpu usage.
Let say, at first the average response times are 300ms. After 2 days, the response times are now 600ms, which I don't understand.
Of course I did some checks:
There are no memory leaks (confirmed with jprofiler)
Heap mem is always around 25-50%, perm space usage is 50%
GC is almost never busy
All threads are idle after inspecting a thread dump
While do some further investigations, I did a cpu profile with JProfiler at the beginning (when it's still fast), and on on the (slow) end. What I see then, is that every single call just is 100% slower!
Even call's to a simple Map#put(). (the # of invocations and the content of these maps are the same).
When running a profiler, there are no signs of blocked threads, just running threads.
Does anyone has a clue what's causing the performance degradation?
Thanks!
Update: solved the performance degradation by upgrading the Java version to 1.6.0_24 !
While out of options, I scanned through all the release notes of the java vm, and discovered a performance and reliability fix in 1.6.0_23. See also the
1.6.0_23 release notes
After the jvm upgrade, the performance stays the same and does not degrade over days.
Solution found by Jan :
Solved the performance degradation by upgrading the Java version to 1.6.0_24 !
While out of options, I scanned through all the release notes of the java vm, and discovered a performance and reliability fix in 1.6.0_23. See also the 1.6.0_23 release notes
After the jvm upgrade, the performance stays the same and does not degrade over days.*
Related
It seems to me that SBT is taking an excessive amount of CPU time just watching for file changes. I'm aware of this post, but the author there was confounding the IDE cpu time with SBT cpu time; I don't have an IDE running.
I'm developing a Play project in Scala with about 370 scala files divided up into 5 modules.
Running sbt with ~run consumes about 70-90% CPU on my MacBook Pro (mid 2012). I don't have an IDE running, I'm not editing any files, no browser accessing the server... Just idling with ~run (and watching the file system) takes 70-90% of a CPU according to Activity Monitor.
After watching it idle for a while I fired up VisualVM; it shows itself and one other JVM process:
Details on the xsbt process (per VisualVM):
PID: 56661
Host: localhost
Main class: xsbt.boot.Boot
Arguments: -Dhttp.port=9001 -Dhttps.port=9443 -Djsse.enableSNIExtension=false -Dhazelcast.config=conf/hazelcast-dev.xml -Dlogger.file=conf/logback-dev.xml -Dconfig.file=conf/passwords/local-dev.conf
JVM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13, mixed mode)
Java: version 1.8.0_121, vendor Oracle Corporation
Java Home: /Library/Java/JavaVirtualMachines/jdk1.8.0_121.jdk/Contents/Home/jre
JVM Flags: <none>
Heap dump on OOME: disabled
JVM arguments:
-Xms1024m
-Xmx1024m
-XX:ReservedCodeCacheSize=128m
-XX:MaxMetaspaceSize=256m
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005
Here's a bit more detail from VisualVM on where the time goes:
Is 80% CPU usage -- and fans running most of the time -- reasonable while sbt is idling? Seems really high. Any strategies for bringing it down?
PS: When sbt is just at the command prompt (not watching for file changes) Activity Monitor shows about 5% CPU usage.
PPS: I normally have concurrentRestrictions in Global += Tags.limit(Tags.Test, 4) in my build file to prevent too many concurrent database connections during tests. I'm not sure if this actually does what I think it does. The stats, above, are with that line commented out. When I restore it Activity Monitor still reports ~80% CPU usage but in the CPU samples idleAwaitWork doesn't show up at all and ConcurrentRestrictions$$anon$4.take takes the top spot with SourceModificationWatch$.watch right behind it.
Assuming you're using sbt 1, this might be due to https://github.com/sbt/sbt/issues/3860 as pointed out by Waldemar Wosiński in comment.
Since we've adopted Close Watch, the problem should now be fixed. Please try sbt 1.1.6 or later and see if the problem still persists.
recently I had to run few heavy one-time queries on our MSSQL 2008 R2 64-bit server and faced a problem: executing them made SQL server consume 100% CPU which eventually (in about 20 seconds) made server absolutely unresponsive.
Thus I was forced to reboot it or wait until execution completes which took a lot of time depending on a query.
What I noticed is that setting CPU Affinity for SQL server to 7 cores instead of 8 available in task manager would keep server responsive so I could cancel my query if it took too long (and proceed with query optimizations without having too reboot).
But is it a good idea to limit CPU Affinity of SQL server?
Please share your thoughts. Server is being used for web-applications.
It turns out to be a Bad Idea.
After few days with CPU affinity 7/8 I noticed that SQL server would continuously load 1-2 cores up to 100% while other cores were available.
It is probably true that SQL Scheduler cannot distribute workload correctly when CPU affinity is limited.
Its years later but in case anyone finds this in search, your assumption is correct that work schedulers become locked to a core. However there is a trace flag to turn on in order to put this back: 8002.
Recently have been experiencing a strange problem on our JBoss5. After running our app for a while, the clients who call the EJB's start Throwing NoClassDefFoundError on some classes. After a restart, all is fine again for a while until other functions start returning NoClassDefFoundError. It seems totally random and a restart of the JBoss seems to cure the problem. This particular JBoss runs in a VM with 4GB of RAM and 2 CPU's and more than enough disk space (it has never has less than 5Gb free at any time). We have increased the Xmx and XMs to 2048 Mb and the permgen sweeping to 512Mb (ridiculousness I know). Intersetingly, the same install runs elsewhere on a VM with half the memory and Xmx/Xms/permgen settings with no problems whatsover. The Only differnce being that the last stable one is not any major load , although the broken one only has maximum of 8 clients connecting which could hardly constitues "load" in my books :-). Has anybody come across this kind of problem, or have any idea of what it could be?
Not really an answer, be we had a RPM install for CENTOS 5. We removed that and used the zip file from the the jboss site instead. That cured the problem. Looks like we had a Dodgy install.
At work we are using some pretty powerful machines: HP Z600 with dual xeon #2.5GHz, 8-16GB ram. Unfortunately due to ill-fitting company policies we are forced to use 32-bit XP, so I made a PAE ramdrive out of the unused 4GB RAM.
Now, the temporary files are on the ramdrive. I've also tried moving the whole project to the ramdrive, then to an SSD, but there was no noticeable improvement in hosted-mode startup or compilation times.
I've then run SysInternals' Process Monitor to see if there are any bottlenecks that are not visible with task manager / hard-disk activity led, but I have not seen anything notable - except some buffer overflows which I don't understand what they mean.
I can assume that performance for OOMPH startup and GWT compilation are tied so I'm using the compilation times as benchmarking between various changes.
I've activated and deactivated hyper-threading and turbo-boost in BIOS, but again saw no differences. Hyperthreading even seems to make everything slower, I can assume that context switching penalty is higher for 16 cores than for 8 cores. Turbo-boost does not seem to do anything, I can assume it only works under Win7, I have not succeeded in activating the driver. It should boost the core from 2.5Ghz to 2.8Ghz.
Deactivated indexing and timestamping on NTFS drives, changed the performance setting from foreground to background and back, used another instance of Eclipse - no changes.
For compilation I've tried specifying a different number of workers, larger memory and some other options. Everything above two workers increases compilation time.
Older HP machines (XW6600) seem to compile a bit faster, perhaps because of the 2.8GHz clock, but their hosted mode seems to start up slower.
To sum up, memory usage is at about 2.6GB, pagefile usage is zero, harddrive is not signaling much activity, CPU activity is <10% (single core at about 50-70%) but still the computer seems to do nothing for some time while compiling or launching OOMPH GWT.
Ok, so now that I've tried everything I knew and found on the Internet, is there anything else that I can try? Will switching to 64-bit Win7 improve much (this is due next year anyway)? Are there any hardware/software options that I can tweak?
L.E.: also ran RATT (tracer from MS) to see if there are any interrupts taking too long, but everything seems in order. Antivirus does not make a difference. Benchmarked another GWT project against my i7 mobile (2630q) and the i7 is about 70% faster, though it has about the same clock.
In most cases I've encountered the reason for slow compilation/hosted mode refresh is that application is written this way.
For compilation there are few things to watch out for:
Don't keep unused classes and modules in your classpath, remove them if possible because GWT is parsing them anyway at precompile stage
whatch out for GWT-RPC type explosion, this is usually a cause of all problems
Be really carefull with interface ContextWithLokup, always try to minimize the number of methods used in interface which extends ContextWithLookup
Try to check out some other compilation options, like distributed compilation , soft permutations or multi-jvm compilation (run compiler with system property -Dgwt.jjs.permutationWorkerFactory=com.google.gwt.dev.ExternalPermutationWorkerFactory and -localWorkers to specify number of JVMs)
For hosted mode:
Use lazy loading where it is possible. The greatest problem with hosted mode i saw, i that application is trying to initialize too many classes which aren't even used on current screen, this can greatly speedup a hosted mode startup. Again, stuff like RPC type explosion affects hosted mode as well
That's all I can advise. Stuff like ram disk can speed up stuff like -compileReport, since it can generate a huge number of files.
Our ASP.Net 2.0 web app was running happily along on Windows Server 2003. We were starting to see some of the limits of the environment approaching, such as memory and CPU usage spikes, and as we're getting ready to scale we decided it was time for a larger server with higher availability.
We decided to move to Windows Server 2008 to take advantage of IIS 7's shared configuration. In our development and integration environments, we reproduced the OS and app in 2008/IIS 7 and everything seemed fine. But truth be told, don't have a good way of simulating production-like loads as of yet, nor can we reproduce our prod environment accurately (we're small with limited resources). So once we rolled out to production, we were surprised to find performance significantly worse on 2008 than it was on 2003.
We've also moved from a 32-bit environment to 64-bit in the process, and we've also incorporated ASP.Net 3.5 dll's into the project.
Memory usage is through the roof, but I'm not as worried about that. We believe in part this is because of the overhead with Server 2008's memory, so throwing more RAM at it may solve that issue. The troubling thing is we're seeing processor spikes to 99% CPU Utilization, which we've never seen before in the 2003/IIS 6 environment.
Has anyone encountered these issues before and are there any suggestions for a solution/places to look? Right now we're doing the following:
1) Buying time by adding memory.
2) Buying time by setting app pool limits: shut down w3wp.exe when CPU hits 99% load. Since you don't have the option to recycle the app pools, I have a scheduled task running that recycles any stopped app pools.
3) Profiling the app pools under Classic and Integrated modes to see which may perform better.
Any other ideas are completely welcome.
Our experiance is that code runs much faster on a 64bit windows 2008 than on a 32bit windows 2003 server.
I am wondering if something else is also running on the machine. For example is SQL Server installed with a maintainence plan that could cause the CPU spike.
I would check the following:
Which process is using the CPU?
Is there a change in the code? Try installing the new code on the old machine
Is it something to do with the compile options? Is the CPU usage a recompile?
Are there any errors in the event log?
In our cases, since we have 4 processors, we then increased the "number of worker process to 4" currently working well so far as compare before.
here a snapshot:
http://pic.gd/c3661a
You can use the application pool "Recycle" option in IIS7+ to configure physical and virtual memory limits for application pools. Once these are reached the process will recycle and the resources will be released. Unfortunately the option to recycle based on CUP usage has been removed from IIS7+ (some one correct me if I'm wrong). If you have other apps on the server and want to avoid them competing for resources when this condition happens you can implement Windows System Resources Manager and it's IIS policy (here is a good tutorial http://learn.iis.net/page.aspx/449/using-wsrm-to-manage-iis-70-apppool-cpu-utilization/)
Note SRWM is only available on Enterprise and Data Center editions.