We are planning to migrate our enterprise application currently running on Java5 stack to Java7 stack. We are having issues with implicit gc calls (mainly major gc) causing system to be unstable for a short time(ranging from 5 mins - 30 mins). After analyzing the gc stats, we found that Compact phase is taking quite long time to complete when compared to Mark and sweep phase. I understand compaction is quite complex and time taking but its impacting the app server which is customer facing and few connections being dropped off during this phase.
Now, my question is as we are migrating to Java7, is there a better garbage collection process compared to Java5?
App servers are provided with decent system resources.
Each app server contains 32 cpu cores
contains 64 gb ram
App server is IBM webpshere server
Operating System - 64 bit IBM AIX
As said earlier, gc is happening because of implicit system calls. No explicit system calls invoking gc.
Now, my question is as we are migrating to Java7, is there a better garbage collection process compared to Java5?
Generally yes, though as #Pushkar, you should really be migrating to Java 8.
With respects to the specifics of your application(s), it sounds like you need to tune / retune the garbage collection on Java 5. If you are periodically experiencing 5 to 30 >>minutes<< of unstability due to GC, there is something rather wrong. The current behavior may be due your application or Websphere (e.g. memory leaks, excessive caching, etc), or it may be due to poor GC tuning.
In short, switching to Java 7 (or 8) might make things better "out of the box", but it is likely that you will need to put in more effort to address the underlying cause of your problems.
Finally, I'd advise the obvious things.
Implement the changes in small steps. Don't upgrade your app, websphere version, java version, etc all at the some time.
Do the upgrades of your servers one at a time. Have a roll-back plan in case you get unacceptable performance.
If possible test it all first ... including performance / load testing.
By default, java 7 uses parallelGC on server class machines. If you are using JDK 7 update 4 or later version, switch to G1 garbage collector which might give you better performance. But as #the8472 suggested,it will be good to know what settings you used in java 5 and now in your current environment.
Java 7 reached end of life around April 2015. Why not migrate to 1.8?
GC performance usually improves with java major releases (and in some cases with minor GCs).
You should take a look difference GC tuning flags, following link may help you
http://www.oracle.com/technetwork/articles/java/vmoptions-jsp-140102.html
http://stas-blogspot.blogspot.com/2011/07/most-complete-list-of-xx-options-for.html
Related
We have a large JDK7 application deployed on JBoss using several libraries like Hibernate, Spring and so on. After initial startup of the server, the application runs as expected but after some uptime it becomes very slow.
Using a profiler, we have seen that every time certain aspects of out application are slowing down but not always the same aspects. While in one run it might be that hibernate flush slows to a crawl, in another run it might be some DI-code from Spring.
What's going on there?
There is a bug in JDK7 regarding the CodeCache memory area which hit us very, very hard.
Explanation
Basically Java starts up and uses just in time compilation (JIT) to compile just the required parts of the bytecode during runtime. This enables the JVM to de- and recompile certain code fragments during execution. This happend, if the JVM determins, that an initial compilation of a certain code fragment is suboptimal. Oracle introduced a feature named tiered compilation in JDK 7 which allows the VM to do just that.
Compiled code in the JVM is stored in the CodeCache memory area. Up to JDK6 the default was that this area would be filled up and once at a 100% the JIT would stop compiling and an error would be printed to the console, however the application would be running same as before: Everything already compiled would stay compiled, everything not yet compiled would be executed in interpretation mode (which is roughly 100x slower)
This option is named CodeCacheFlushing, it is enabled by default since JDK7u4. The idea is, that once CodeCache is full, the least used parts of compiled code are flushed from memory to make room for other code fragments. That would make the JDK6-default-behaviour (to stop compilation all in all) obsolete. It also allowed for a much smaller CodeCache area (in JDK7 CodeCache is 48M by default/96M if tiered compilation is enabled).
Here comes the bug. In JDK7 once the CodeCache gets full, the JIT is stopped. Next comes the flushing of the CodeCache area. That's it. JIT should be reenabled after flushing is completed but that doesn't happen. Also, there is no warning printed to the console. Worse: prior to disabling the JIT roughly half of the already compiled code is thrown out.
In contrast to JDK6 where everything that was fast will stay fast and only new code will be interpreted, in JDK7 you actually lose already compiled and optimized code! All of the sudden parts of your application that performed well will stop doing so. It is left to chance, which parts of the application slow down, which makes tracking that bugger by profiler nearly impossible: At times the hibernate code for flushing slows down, at other times, its the spring DI code or your own appcode.
Are you affected?
You can use a profiler (JProfiler/YourKit) or JConsole (JVisualVM won't do) to monitor the memory-consumption of CodeCache memory area. Typically the CodeCache amount committed will stay very close to the used amount (say, committed is 23mb, used is 22mb). While your application runs, committed and used go up until committed reaches max. At that point used will drop sharply to 1/2 - 2/3 of max. After that, used whill no longer grow. That's where the bug will hit you. In JConsole, it will look like this:
Why me and not all the others?
Chances are, you are using JBoss. Oracle quickly found out that there are things not like they should be and disabled tiered compilation by default - yet Red Hat in its infinite wisdom decided, it knew better and reenabled it. Basically our webapp runs fine on Weblogic and only JBoss is affected, because without the tiered compilation (not enabled in weblogic) the growth of CodeCache is so small, we never actually hit the 48mb threshold even after weeks of operation.
What can I do?
First of all, decide, whether this bug hits you. Second, make it harder for the bug to damage you. If you disable CodeCacheFlushing at least hitting the bug won't make things worse than they were before. Stopping tiered compilation will make it less probable the bug hits you, same as increasing the amount of CodeCache-Memory available.
You can always try to switch to JDK8, this seems unaffected and also you could implement monitoring in your software to warn you, if CodeCache is running full.
TL;DR
In JDK 7 never enable tiered compilation (disabled by default, enabled in JBoss)
in JBoss 7 always set PRESERVE_JAVA_OPTS=true in standalone.conf
always disable CodeCacheFlushing (-XX:-UseCodeCacheFlushing)
always pack a sufficient amount of memory into CodeCache (-XX:ReservedCodeCacheSize=xxM).
While developing Java EE applications, the NetBeans IDE (currently using 8.0.2 but the issue is not restricted/limited to this version only) issues a low memory message quite frequently.
For example, while modifying and saving some Java classes (especially, JPA entity classes), the memory soon gets over-flooded after repeating this modifying and saving process for merely 10 to 15 entity classes. The IDE issues a low memory message disturbing the whole system severely.
It leaves only one alternative which is to restart the system and I indeed have been wasting more time in restarting the system than actually developing Java EE applications for couple of years :). I have not yet seen anything about this issue over the internet.
The auto-deploy (Deploy on save) option on the IDE is turned off forever. I never use this facility as it causes the heap/PermGen space to get over-flooded quite soon.
This may or may not happen in small toy applications. I am only talking about Java EE applications using a Java EE compliant application server.
I see if there is no way to get around this problem, then this IDE cannot be used in developing real Java EE applications because dealing with this problem basically takes more time than the actual development time and otherwise, the solution may be to lean towards other IDEs like Eclipse, IntelliJ Idea, JDeveloper etc, if they go better.
I am currently using,
GlassFish Server 4.1/Java EE 7
JDK 8u45
NetBeans 8.0.2
Is there a solution somewhere in the world?
NetBeans tends to be slower than other IDEs on Windows but this is far beyond the single term slow.
The picture is taken from a simple test web application (thus not an enterprise application). In enterprise applications, changing a single character in a single entity class is not affordable. One is likely to run into a big problem, if attempted. Fixing those errors shown in the link may take hours or probably the whole day or so.
Not to mention again that I am not talking anything other than large scale applications involving Java EE or other platforms like Spring.
Long story short : Nothing can prevent me from wondering as to how it is affordable to use this IDE in software industries. It is no longer usable in this way (Nothing to abuse. Honestly, I kept patience for almost three years. I am merely wasting time using this IDE).
I get that Postgres scales automatically to multicore with multiple connections, but what about when I'm running a massive query on a SINGLE connection? So frustrating that the CPU usage maxes out at 25% on my 4-core system.
I'm in process of switching from SQL Server and this is the only thing so far that really bugs me. SQL Server will use up to 100% of my CPU for a single connection/query.
I'm running 9.2 on Windows 7 Enterprise 64-bit with Xeon processor if it matters.
If there is not way to get around this, could someone address why this isn't seen as an issue? Is it because Postgres is focused on multi-user scenarios?
PostgreSQL does not currently support executing a single query across multiple CPU cores (minus background things like background writing and wal writing if you're doing a write query, but that doesn't really count). It's work that's in progress, but it's a long-term project, and is not in any current version of PostgreSQL.
This is the same on all platforms and architectures.
It is definitely an issue, but since PostgreSQL is, as you say, focused on multi user scenarios, it's not bubbled to the top of the priority queue until recently. But there are definitely people realizing it's an issue, and working on solving it for future versions, it's just not done yet.
There is a Foreign Data Wrapper that aims to add parallelism via the GPU called pg_strom. I've never used it, and it looks quite specialized, but maybe you (or someone here) has a use-case for it.
Article describing pg_strom
http://gpuscience.com/software/postgresql-gpu-pgstrom/
The code:
https://github.com/kaigai/pg_strom
It's not that it isn't seen as a problem. It's that it requires fundamental architectural changes. The use case for it is pretty specialised. It would only help on data warehouse type environments where you're executing long queries one at a time -- AND the queries are CPU bound, not disk i/o bound as they would usually be.
What does a Profiler do exactly?
I ran the JProbe profiler on my Java EE Application.
For now I selected Performance Analysis only. When I investigated the code, it showed how many times each method gets called and how much time it took. It gave me a clear view of these things.
Now my question is, what in general does a profiler do exactly? The only thing it seems to do is showing how many times a method is called and how much time each method took?
Does Profiling a Java EE application indeed only means this thing? (In the concern of Performance Analyses only)
A profiler can tell you lots of useful things, in addition to traces and method timings:
The state of the heap and its generations in real time: perm, eden, etc.
Created threads and their states
CPU usage
Number of instances of each class
I like to use Visual VM 1.3.3 with all the plugins installed. I use the Oracle/Sun JVMs, so it works for me.
Just to add..Profiling is general concept not only restricted to Java or Java EE.
It is a dynamic analysis of your program.
what does it do? you can check on profiler application.
how does it help: Helps you to optimize your program and trouble shoot cases such as Out of memory and dead lock which is not possible with the routine debugging techniques (printlns and debugger).
How can I improve performance when developing locally with Websphere and RAD? I am using one web application of moderate size (1000? classes) and it is impossible to handle the app locally on a Windows box. The Websphere 6.1 configuration uses the default configs. RAD7 is configured to handle a max heap of 1024mb. I thought about increasing the heap of the server. At present, the min and max are 128/300mb.
In terms of unresponsiveness, sometimes it may take minutes to load a page, if the page loads at all. Also, I disabled "Build Automatically" and Publish Automatically. Maybe those should be turned on?
I'm not sure about RAD7 but from my past experience, I'd suggest to give MyEclipse Blue a try.
Since that might not be an option, here are some other usual culprits, you can check:
How much RAM does your machine have? It's good to give WS 1GB of RAM but if your computer only has 1GB of real RAM, it's going to swap itself to death. If your boss won't pay for it, go get some RAM with your own money. 2GB are less than $80 ATM. I suggest to get at least 4GB. Yes, Windows can only use 3.5GB even when 4 are installed but that half GB costs $20 or less. Even thinking about this for more than five minutes will cost more than simply buying it.
Next make sure whether you are using the correct Java GC options. There should be some info about this in the docs. Plus make sure that the process uses the "jvm.dll" from the "server" directory, not the "client" one. "Process Explorer" will help.
Since I'm not using RAD, I'm not 100% sure about "Build Automatically" and "Publish Automatically" but since RAD7 is based on Eclipse, these options will compile code in the background as you type. This will greatly reduce the time between you saving your last change and the moment the app server can start to load the new code.
When all else fails, run websphere in a profiler and look where it spends all the time.
Aaron had great advice.
I would also suggest using JConsole to see what is going on, to help you determine if you need more memory, larger heap size, etc. My experience with running Websphere and RAD locally is that it will be slow, but then I was on an old machine that needed more memory. :)
http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html
Berlin,
RAD 7 saps your PC! When I was using it to develop Portlets, I followed this optimization guide and it made the IDE significantly quicker to develop Portlets in. Obviously it is aimed at Portlet development but it might help you.
Also following the advice given to the answer to this question will also help.
I definitely agree with Aaron Digulla. You will see a major performance improvement with 4GB RAM installed on your development machine. I developed an Eclipse/RAD plugin with some buddies of mine and we were able to measure how much time we saved by upgrading from 2GB to 4GB.
The plugin is available here: http://lopb.org/
After gathering some hard numbers on how much time we spent waiting for publishing and loading the app on our 2GB development machines, we were able to convince management to upgrade the rest of the developers on the team.
Anyway, you should really consider upgrading to 4GB if you want to run RAD 7 and Websphere 6 on the same development machine. Each one needs -Xms=512m -Xmx=1024m as JVM args to run well, and that means you will swap to disk way too much if you only have 2GB of RAM or less. HTH
Make sure your running was in development mode for your development and testing.
Option is under the server in the console.
Karl
hehe, we had the same problem with RAD6 and Websphere 6.
The way we speeded things up is moved to Eclipse and JBoss.
We developed on Eclipse and JBoss and then first round of testing was on Websphere. We had some issues with the differences but would never had completed the project were it not for out switch (a lot fewer issues than deving on RAD/WAS).
But to help you in the mean time...
Definitely, probably want build automatically and publish automatically off. That way you can make a bunch of changes and then tell RAD to compile and deploy while you go and get coffee.
There is a "run in dev mode" in Websphere (I know there was for 6.0) so track that down and turn it on (it's on the WAS console somewhere)
I found WAS's on stack replacement to work fairly well. I found that at the beginning of the day I'd deploy to WAS and then not have to redeploy at least until lunch time (as I was debugging). I would make changes and the changes would be fed to the server without my having to redeploy.
Chances are, even after running the profiler you'll find there's nothing much that you can do..
Turn off all validations (in RAD), they tend to take forever.
Depending on what you're doing with EE investigate the possibility of deving on another IDE/Server combo, maybe you can do the bulk of your work in there and then deploy from RAD/WAS to do some final testing. If you're using vanilla ejb's or web services this is feasible.
That max heap does sound a bit small to me. The suggestion to fire up JConsole is a good one cause it will tell you how much heap is being used, though I'm not sure if it will work on the IBM vm (RAD's). You might try and turn on the memory usage monitor in RAD, tells you how much memory is being used, that way you can tell if it's hitting the max.
JConsole will not work without specifically enabling it via a JVM command line switch.
Suggestions from Michael Wiles sound reasonable but please update your RAD first to the latest FixPack available.
You can also contact support.