We are trying to load test GWT Application (2.0.4) where all the widgets are loading on a single page and it makes nearly 20 rpc calls while loading the page.
Server: JBoss 5 and we are monitoring the heap memory using JConsole.
During the load test we notice that the heap memory increases to max even for min number of users. However the memory is getting reclaimed once gc is triggered.
Could anyone explain Why the memory utilization (heap memory) is so high for GWT application on server even for small number of users?
We are using Hibernate + Spring framework.
1) It would help to reduce the number of GWT RPC calls on application screen launch by combining the data they fetch. Multiple RPC calls on load tend to block up lot of resource on browser and on server.
2) Analyze whether you need all the data your RPC calls are fetching first up.
3) The heap space usage varies depending on server side technology you are using Hibernate, Dozer, Gilead, lazy/cache loading etc.
4) Code Splitting and Fragment Merge help you reduce your app javascript which you load first up.
I am guessing you are using Dozer to copy hibernate entities to DTO/beans of client side. Try playing around with hibernate C3P0 connection pool settings.
Related
I wrote a sample WebFlux application, which read just some data from redis and do small cpu job(md5 calculation), repeats several time.
It use spring data redis reactive
'org.springframework.boot:spring-boot-starter-data-redis-reactive'
Codes that connects redis is like this.
reactiveStringRedisTemplate.opsForValue().get(keyname)
redis server runs at localhost
redis-server &
Full codes , you can find here
https://github.com/mouse500/redperf
It's a simple codes
And for test, I call API (/wredis) with Jmeter to loadtest
Problem that I think is...
this application doesn't reach maximum TPS,
It reaches CPU around 40% on my local PC.
Even though it has more room of CPU, it doesn't work any harder
Why doesn't it use resource fully?
In case of some other method to connect redis,
(I put connection proxy ,written in nodejs)
It showed much much higher CPU and got much more TPS.
So I don't think it is not about Redis server performance.
Problem seems issue at "calling Redis from WebFlux application with lettuce"
How can I make this sample application show a maximum TPS ( CPU reach 100%) ?
What option can I try?
I created and ran a simple SpringBoot application to accept some requests. I used jconsole to check the Heap Memory Usage and I saw this periodic increase followed by GC, I don't know the reason for the increases. Are there any Objects keep being created (because I think the instances are imported to container when the application starts)?
Spring boot has background processes which may consume your resources despite on absent requests to your app, for example jobs, etc.
Those spikes are expected and regular for any Java app based on any more less complex framework. GC algorithm depends on your jvm version, but could be overridden. The graph shows normal situation, from time to time memory consumed for some activities and after some time GC wake up and do the cleaning.
In case if you want to check what exactly objects caused memory spike you may try to use Java Flight Recorder or regular heap dump analysis using Eclipse memory analyser.
For current case Java Flight Recorder would be more convenient and suitable.
The web application is running on Springboot and deployed on WebLogic.
We have assigned 400 as max threads and JDBC to be 100 connections.
When we perform load testing on the web application, the performance is optimal when the load is low (the response time is less than 200ms for most of the http request that we called).
When we increase the load, we can see that the thread count increases and jdbc count also increases gradually but no where near to max. However, the response time is getting much longer and it could take more than 5 seconds to response.
CPU usage, thread count, memory, JDBC connection seems to be normal during these period.
Another observation is that during testing and we saw that the performance is degrading, we used another machine to make a http call to the server that is only retrieving text without any DB or logic, and even this simple http call will take 10s to respond. (And the server resources is still not MAX!)
So, we are wondering what keep them waiting ?
Any other possible bottleneck?
If the server doesn't lack resources like CPU/RAM/etc. only a profiler can tell you where your application spends the most time which might be in:
Waiting in a queue for next thread/db connection from the pool to be available
Slow database query
Inefficient functions/algorithms which a subject to optimization
WebLogic configuration not suitable for high loads
JVM configuration not suitable for high loads (i.e. system is doing garbage collection to often/too long)
So I would recommend re-running your test with profiler tool telemetry enabled and at the same time monitoring essential JVM metrics using i.e. JMXMon Sample Collector which can be used for monitoring your application-specific metrics as well. It's a plugin which can be installed using JMeter Plugins Manager
For a detailed approach on how ago about identifying poor thread performance I suggest you take look at the TSA Method by Brendan Gregg.
What is the most efficient way of having Infinispan/JBoss Data Grid in library mode with several applications using same caches?
I currently setup JBoss Data Grid in library mode in EAP 6.3, have about 10 applications and 6 different caches configured.
Cache mode is Replication.
Each application has a cache manager which instantiates the caches that are required by the application. Each cache is used by at least 2 applications.
I hooked up hawtio and can see from JMX beans that multiple cache managers are created with duplicated cache instances.
From the logs, I see:
ISPN000094: Received new cluster view: [pcu-18926|10] (12) [pcu-18926, pcu-24741, pcu-57265, pcu-18397, pcu-26495, pcu-56892, pcu-59913, pcu-53108, pcu-34661, pcu-43165, pcu-32195, pcu-28641]
Does it have a lot of overhead in cache managers talking to each other all the time?
I eventually want to setup 4 cluster nodes with JBoss data grid in library mode so how can I configure so that all applications in one node share same cache manager hence reducing noise?
I can't use JBoss data grid in Server mode which I am aware will fulfil my requirements.
Thanks for any advice.
First of all, I may misunderstand your setup: this log says that there are 10 'nodes'. How many servers do you actually use? If you use the cache to communicate with 10 applications on the same machine, it's a very suboptimal approach; you keep 10 copies of all data and use many RPC to propagate writes between the caches. You should have single local-mode cache and just retrieve a reference to it (probably through JNDI).
Cache managers don't talk to each other, and caches do only when there is an executed operation, or when a node is joining/leaving/crashing (then the caches have to rebalance).
It's JGroups channel that keeps the view and exchanges some messages to detect if the other nodes are alive or other synchronizing messages, but this kind of messages is send once every few seconds, so this has a very low overhead.
On the other hand, each channel keeps several threadpools, and cache manager has a threadpool as well, so there is some memory overhead. From CPU point of view, there is a thread that iterates through the cache and purges expired entries (the task is started every minute), so even with idle cache full of entries some cycles are consumed. If the cache is empty, this has very low consumption (there's not much to iterate through).
I'm working in financial sector and we are about to select Vaadin 7 for development of large heavy load system.
But I'm a bit worried about Vaadin memory footprint for large systems since Vaadin keeps all state in session. It means that for every new user all application state will be stored in memory, won't it?
We cannot aford to build monolithic system - system must be scalable and agile instead. Since we have huge client base it must be easy to customize and ready to grow.
Could anyone please share the experience and possible workarounds how to minimize or eliminate those problems in Vaadin?
During the development of our products we faced the problem of large memory footprint using the default Vaadin architecture.
The Vaadin architecture is based on components driven by events. Using components is fairly simple to create a tightly coupled application. The reason is that components are structured into a hierarchy. It's like a pyramid. The larger application is built; the larger pyramid is stored in the session for each user.
In order to significantly reduce the memory allocation we've created a page based approach for the application with a comprehensive event model on the background using the old school state management. It is based on the Statechart notation in XML format.
As the result the session keeps only visited pages during the user workflow, described by the Statechart configuration. When the user finishes the workflow, all the pages are released to be collected by garbage collector.
To see the difference we have done some tests to compare memory allocated for the user working with the application.
The applications developed:
with tightly coupled approach consume from 5 to 15MB of heap per user
with loose-coupled approach - up to 2 MB
We are quite happy with results since it let us scale the large system using 4GB RAM up to 1000-1500 concurrent users per server.
Almost forgot. We used Lexaden Web Flow library. It is with Apache license.
I think you should have a look here: https://vaadin.com/blog/-/blogs/vaadin-scalability-study-quicktickets
Plus, I have found the following info by people who run Vaadin in production.
Balázs Hódossy:
We have a back office system with more than 10 000 users. The daily
user number is about 3000 but half of them use the system 8 hours
without logout. We use Liferay 6.0.5 Tomcat bundle and Vaadin as
portlet. Our two servers have 48 GB RAM and we give Tomcat 24 GB heap.
DB got 18 GB and the system the rest. Measure the heap to the session
size, concurrent users, and the activity. More memory cause more
rarely but longer full GC. We plan to increase the number of Tomcat
workers and reduce the heap. When you measure your server, try to add
a little bit more memory. If the cost is so important than decrease
the processor cost and buy more RAM. Most of the time it is valuable
with a little tuning.
Pierre-Emmanuel Gros:
For 1000 dayly user heavyly used , a pure vaadin application: Server
3 gb 2 core Jetty with ulimit to 50000 Postgresql 9 with 50
concurent users ( a connection pool is used). As software part, I used also ehcache to cache DTO objects,and pure JDBC.