I have 2 identical servers and currently I am monitoring both of them by setting up performance counters.
IIS 6 with .NET Framework 2
I am noticing that one server has high disk writes of ~3300 Writes/Sec and the other server has ~199 Writes/Sec
Anyone encountered the same problem?
What may cause high disk writes?
These servers are load balanced (50%-50%)
Thanks
Lots of things can cause high disk activity.
Swapping
There's lots of stuff being written to disk
You left ODBC tracing on (oops!)
...
Sounds like you're already using Performance Monitor; add some more counters to watch per process rather than systemwide:
Process | IO Writes / sec
Process | IO Written Bytes / sec
Process | Page File Writes / sec
I'm not sure about the Page File Writes per second (this is from memory) but there should be something like that in there. You should be able to isolate the high activity to a process and that should help you figure it out.
Multiple issue can cause high disk writes, one of them is paging.
is one of these server under high memory load
Related
Recently, I met a performance issue. I have a program running on redhat 7.5. My server has 1TB memory and 32 cores. vm.min_free_kbytes has set very small to avoid kswapd thread running. My program is like a message queue, it stores the received data to disk frequently, and it will only use 200GB physical memory.
We know that, Linux uses page cache(memory) to improve the performance of IO. when my program writes data to disk, data will be written to page cache(memory) first, then write syscall returns. After running for a long time, the page cache uses all the remaining memory(1TB-200GB=800GB). At this time, the issue is that any threads of my program may be blocked for 1~3 seconds. I have check the sar -B output, the pgscand/s becomes 20000 at that moment.
My question are:
(1) why my program was blocked for so long time(1 - 3 seconds)? It affected RTT in my system.
(2) What is the theory of page cache reclaim?
(3) Was my program switched to uninterrupted state(state D in top) during page cache reclaim? I didn't see it in top output.
(4) if two threads A and B apply for pages, A triggers direct reclaim to release memory, will B block until A's reclaim is done?
We have a spring boot application that runs on 20 servers and we have a balancer that redirects the requests to our servers.
Since last week we are having huge problems with CPU usage (100% in all VM's) almost simultaneously without having any noticeable increase in the incoming requests.
Before that, we had 8 VM's without any issue.
In peak hours we have 3-4 thousand users with 15-20k requests per 15 minutes.
I am sure it has something to do with the heap usage since all the CPU usage comes from the GC that tries to free up some memory.
At the moment, we isolated some requests that we thought might cause the problem to specific VM's and proved that those VM's are stable even though there is traffic. (In the picture below you can see the OLD_gen memory is stable in those VM's)
The Heap memory looked something like this
The memory continues to increase and there are two scenarios, it will either reach a point and after 10 minutes it will drop on its own at 500MB or it will stay there cause 100% CPU usage and stay there forever.
From the heap dumps that we have taken, it seems that most of the memory has been allocated in char[] instances.
We have used a variety of tools (VisualVM, JProfiler etc) in order to try to debug the issue without any luck.
I don't know if I am missing something obvious, or something else.
I also tried, to change GC algorithm to G1 from the default and disable hibernate query cache plan since a lot of our queries are using the in parameter for filtering.
UPDATE
We managed to reduce the number of requests in our most heavily used API Call and the OLD gen looks like that now. Is that normal?
We are trying to implement a web application with Apache Storm.
Applicationreceives a huge load of ad-requests (100 TPS - a hundred transactions / second ),makes some simple calculation on them and then stores the result in a NoSQL database with a maximum latency of 10 ms.
We are using Cassandra as a sink for its writing capabilities.
However, we have already overpassed the 8 ms requirement, we are in 100ms.
We tried to minimize the size of buffers (Disruptor buffers) and to well balance the topology, using the parallelism of bolts.
But we still in 20ms.
With 4 worker ( 8 cores / 16GB ) we are at 20k TPS which is still very low.
Is there any suggestions for optimization orare we just reaching the limits of Apache Storm(limits of Java)?
I don't know the platform you're using, but in C++ 10ms is eternity. I would think you are using the wrong tools for the job.
Using C++, serving some local query should take under a microsecond.
Non-local queries that touch multiple memory locations and/or have to wait for disk or network I/O, have no choice but taking more time. In this case parallelism is your best friend.
You have to find the bottleneck.
Is it I/O?
Is it CPU?
Is it memory bandwidth?
Is it memory access time?
After you've found the bottleneck, you can either improve it, async it and/or multiply (=parallelize) it.
There's a trade-off between low latency and high throughput.
If you really need to have high throughput, you should rely on batching adjusting size of buffers bigger, or using Trident.
Trying to avoid transmitting tuples to other workers helps low latency. (localOrShuffleGrouping)
Please don't forget to monitor GC which causes stop-the-world. If you need low-latency, it should be minimized.
If my application is allotted 500MB of RAM, but operates on 2GB of data, how will this work? What would be the impact on execution speed?
Don't know - depends on what you're doing and how your application operates on data.
You can't fit 10 pounds of anything into a 5 pound bag. You'll have to page or operate on a stream or something.
If you exhaust the memory available to you, you're likely to see an OutOfMemoryError.
Not enough information to give a real answer.
Every time it needed to access data that is outside the current 500 megabyte window, it would need to swap out to disk. That swapping process takes some time, depending on disk speed, and would potentially slow down your application.
Our application is memory intensive and deals with reading a large number of disk files. The total load can be more than 3 GB.
There is a custom memory manager that uses memory mapped files to achieve reading of such a huge data. The files are mapped into the process memory space only when needed and with this the process memory is well under control. But what is observed is, with memory mapping, the system cache keeps on increasing until it occupies the available physical memory. This leads to the slowing down of the entire system.
My question is how to prevent system cache from hogging the physical memory? I attempted to remove the file buffering (by using FILE_FLAG_NO_BUFFERING ), but with this, the read operations take considerable amount of time and slows down the application performance. How to achieve the scalability without sacrificing much on performance. What are the common techniques used in such cases?
I dont have a good understanding of the WinXP OS caching behavior. Any good links explaining the same would also be helpful.
I work on a file backup product, so we often run into similar scenario, where our own file access causes the cache manager to keep data around -- which can cause memory usage to spike.
By default the windows cache manager will try to be useful by reading ahead, and keeping file data around in case its needed again.
There are several registry keys that let you tweak the cache behavior, and some of our customers have had good results with this.
XP is unique in that it has some server capability, but by default optimized for desktop programs, not caching. You can enable System Cache Mode in XP, which causes more memory to be set aside for caching. This might improve performance, or you may be doing this already and its having a negative side effect! You can read about that here
I can't recommend a custom memory manager, but I do know that most heavy weight apps do there own caching (Exchange, SQL). You can observe that by running process monitor.
If you want to completely prevent cache manager from using memory to cache your files you must disable both read and write caching:
FILE_FLAG_NO_BUFFERING and
FILE_FLAG_WRITE_THROUGH
There are other hints you can give the CM, (random access, temporary file) read this document on Caching Behavior here
You can still get good read performance even if you disable caching, but your going to have to emulate the cache manager behavior by having your own background threads doing read aheads
Also, I would recommend upgrading to a server class OS, even windows 2003 is going to give you more cache manager tuning options. And of course if you can move to Windows 7 / Server 2008, you will get even more performance improvements, with the same physical resources, because of dynamic paged/non paged pool sizing, and working set improvements. There is a nice article on that here
Type this down in a notepad and save it as .vbs file. Run it whenever u realize the system RAM is too low. The system cache gets cleared and adds up to RAM. I found it else where on net and giving it here so that it might help you. Also, it is suggested to care that the first record should not ever exceed half your actual RAM. So if u have 1 gb ram, start with the following text in your vbs file.
FreeMem=Space(240000000) <This one is to clear 512 MB ram>
FreeMem=Space(120000000) <This one is to clear 256 MB ram>
FreeMem=Space(90000000) <This one is to clear 128 MB ram>
FreeMem=Space(48000000) <This one is to clear 64 MB ram>
FreeMem=Space(20000000) <This one is to clear 52 MB ram>