We have a Spring-boot REST application running on 3 production machines. A recent update from Spring-boot 2.1.8 to 2.2.2 has shown an initial increase of CPU by at least double. This load then increases over time whereas the old version stays steady.
I have managed to narrow this down to 2.2.x as building with 2.1.11 is ok, but 2.2.0 shows the problem.
To give an idea of scale, the old version stays at around 6% regardless of load, whereas the new version starts at around 15% and gradually increases to over 100% after about 10 hours.
I can see the initial rise with an identical build, only changing the Spring-boot version. The application uses spring-boot-starter-web and spring-boot-starter-actuator.
Any ideas? Should I raise this over at https://github.com/spring-projects/spring-boot/issues ?
This is very likely to be linked to a bug in Spring Framework that was fixed in Spring Framework 5.2.6 (or Spring Boot 2.2.7). There was a memory leak in case of concurrent requests/responses with the same media type.
See the dedicated issue as well as a report sent by a developer with lots of details. Note that this happens with both MVC and WebFlux.
We've seen this issue in some of our services, but upgrading to 2.2.7 looks to have resolved this for one (stable for two weeks).
We're starting to roll this out to more services in the hope that it can be rolled out everywhere, so might be worth trying this out?
Related
According to netty documentation, default reactor.netty.ioWorkerCount count is max(4, number of cores), which seems true on local environment. I have a 6 core laptop, and the number of reactor-http-io threads were 6.
But on deploying docker image in kuberenetes we found that reactor-http-epoll (linux) thread count was 36. Our CPU configuration is: Request 4, limit 6.
This doubt was also raised by #ROCKY in one of the comments in Threading model of Spring WebFlux and Reactor.
It seems like it is still unanswered.
So is there something that explains this behaviour?
I think we have have found the answer. Our machine had 36 cores. But our pod had the configuration of 4 cores. So it seems like netty was picking up the machine configuration and not our pod configuration, So this seems like a bug in netty or something else that we are missing.
We are using Spring boot 2.2.0.RELEASE and reactor-netty 0.9.9 version.
why I have spend some extra effort to migrate my application if I'm not using new features of latest version
You don't have to upgrade, but I'd say it's worth it for 2 reasons:
Performance and security upgrades of Spring and other dependencies is always a worthwhile effort in my opinion. By skipping upgrades you could be using vulnerable packages.
What happens in a year if you do need a new feature or need to migrate to JDK 11 and beyond? It's typically easier to do the incremental updates multiple times per year rather than a big-bang upgrade every couple of years.
I'm having a project using both HBase 1.0.0 (Cloudera version) and Elasticsearch. With the upgrade to ES 2.0 I'm experiencing a problem with guava version. ES 2.0 requires guava version 18.0, but Cloudera requires guava 14.0.1.
No matter what version I define in my dependency management in my parent pom one of the two won't work.
Looking around I see that this problem occurs quite some time (e.g. http://gbif.blogspot.co.at/2014/11/upgrading-our-cluster-from-cdh4-to-cdh5.html)
1) Any ideas on how to solve this problem without any complex re-design of my application?
If not, I'm thinking of doing all the ES-stuff in a separate application. Communicating via messaging (already using AMQ) for indexing. Not sure though how to communicate for search/filter requests (at the moment implemented via Java API).
2) Any other ideas?
3) Any ideas/hints on how to solve the communication issue?
I found this blog post when googling in combination with maven-shade plug-in, so this might be another option.
https://www.elastic.co/blog/to-shade-or-not-to-shade
Our application is having issue with rendering time on various part. We have been able to pin point the performance issue with a profiler (YourKit Java Profiler) within the JSF restoreView phase (lifecycle). Since, we use a lot of components in some pages the rendering of the tree must be the source of the problem. Some question on stackoverflow looks pretty much like what we are dealing. Also, we can see the same slow speed in dev and production.
Some specs:
MyFaces 2.1.7
javax.el 2.1.0
Profiler (CTRL+SCROLL to see details):
We are trying to find solutions. So far, we came up with those:
Use less composition/components (which is not that great, because we liked to make generic component).
We could also go into JSF stateless mode.
Find the actual problem with JSF (IN DEEP).
Recently, we found out that some JUEL library (expression language) could greatly increase the rendering time.
I've been dealing with making JUEL works with Spring 3 (with OSGI). Here's the spec:
JUEL 2.1.3.1 bundle (note: this bundle use a different version of javax.el than the one I use in my application will that cause clash?)
I have found another maven repo which only does Juel Implementation and a more recent version.
Update:
Found a solution for javax.el clashing, since there was a lot of problems in the juel-api (because it's export his own javax.el) I modified the manifest with that line Fragment-Host: javax.el to eliminate it.
Is there a configuration/set up to enable better performance?
Or, how to you actually configure JUEL with MyFaces and Spring?
I would say that the problem is caused by Spring Web Flow. I remember this issue SWF-1540. First check if you are using the right version (2.4.0.M1 or upper). Note personally I have not tested if that configuration has a performance issue, but I can say that MyFaces 2.1.7 an upper versions does not have any issues. It is known that some EL versions has perf issues, so try with juel or with apache EL (jasper-el, bundled with Tomcat).
As a personal opinion, watch out for Spring proxies, it is known that in some cases, when the pages are very big and complex, the proxies has a significant impact over performance. The suggested solution is use Apache OpenWebbeans, which is a very fast CDI implementation, optimized for those cases when you have a lot of beans interconnected.
You can check this article too "Understanding JSF 2 and Wicket: Performance Comparison" for tips about MyFaces performance.
You have to try by yourself what will work for your case, but I hope with these advices you can focus better your search.
i'm at my new job and the projects at work are in spring 2.5 (some portlets). how hard is it to migrate to spring 3 ?
there is a customized portal 1.0 (don't know if this is relevant) and i was wondering if i could just ask to develop in spring 3.
never used spring before and want to learn some, but it would be nice if i could just go for the latest version. got spring in action 3rd edition and would like to make use of that
It principle, it should be a drop-in replacement. There's a bunch of stuff that was deprecated in Spring 3, and huge amount of new stuff added, but nothing was removed.
However, there were a bunch of minor bug fixes, and your code may be unwittingly relying on the buggy behaviour, so some minor adjustments may be necessary, but nothing any worse than that.
Other than that, the big difference is that Spring 3 requires Java5+. I'm assuming you're not stuck on 1.4 :)