Over time the used_memory_rss keeps increasing. I understand the reason behind it getting filled (its how memory allocators work). However I need a solution to free it without restarting redis.
Is there any way to do the same?
in redis 4.0 there is a new command
MEMORY PURGE
that will defragment memory and release it to the OS.
also see
MEMORY HELP
You may reference to this issue compact memory use online
Related
I am running redis on windows and I am having some performance issues. The machine is a Xeon E5 with 32GM RAM and SSD with HW-Raid with Windows Server 2012. There are some other processes running, but they are not critical and are idle most of the time.
I noticed performance problems and operations timeout very often, so I started "redis-cli --intrinsic-latency 100". The output shows that the max-latency goes up to 15000 microseconds, which is very slow I think.
I was also running a memory-profiler: The r/w-performance is not so good (5GB/sec) but I think this should not be the bottleneck. At the moment I have absolutly no idea what to try.
Can you give me some tipps how to find the performance problem?
There is no "fork" as in Linux in Windows. So when you dump your redis db, it can just "stop the world" in order to write on the disk "dump.rdb". Well, they did implement a "Copy-on-write" strategy that don't stop redis, it just copies values when dumping (the redis clients will still be able to get responses from redis). It is in their version log: https://github.com/MSOpenTech/Redis
There is a replacement for the UNIX fork() API that simulates the copy-on-write behavior using a memory mapped file.
This is the real bottleneck of redis in windows as it is an overhead and is more complex (bugs?). It is explained here:http://blogs.msdn.com/b/interoperability/archive/2012/04/26/here-s-to-the-first-release-from-ms-open-tech-redis-on-windows.aspx
As a result you could try running a redis on Linux to test if this is a performance issue of the windows port. Also, the more you write a dump.rdb, the bigger is the overhead (you can change the frequency or try disabling it completely for testing).
Finally, it could also be a network problem and you should check if it is not a network rule / hardware problem (not enough throughput! Bad cable or stuff, firewalls...). Are your redis clients on the same hardware machine?
I have been using a Windows port of Redis called "Memurai". They have a developer edition free of charge.
Now, in one of their blog they claim they have solved the fork() problem. See excerpt below.
Memurai performance seems good to me, even with persistence enabled (both RDB and AOF) although I have not run any specific test myself. There's another blog about Memurai perf in here.
It's worth giving it a try.
"Internally, Redis uses the fork() system call to perform asynchronous writes, but that’s not an option for Memurai because fork() doesn’t exist on Windows. Instead, Memurai uses Windows shared memory to implement a start-of-the-art version of fork() that’s finely tuned for performance and..."
I am not an OS expert, and I am having trouble understanding my server's memory usage. I need your advices to understand the following:
My server has 8 GB RAM and operates as web server. PHP, mySQL and Apache processes consume the majority of the memory. When I issue the command "free" after the system is rebooted, I would normally see something along these lines:
total used free shared buffers cached
Mem: 8059080 2277924 5781156 0 948 310852
-/+ buffers/cache: 1966124 6092956
Swap: 4194296 0 4092668
Obviously, sooner or later the free memory would drop and the cached memory would increase and I assume there is nothing wrong with that since the OS decides to cache it.
What I don't understand is about 1-2 days later after the machine is rebooted, I would slightly see an increase in the used swap memory. Does not this mean that the server does not have free memory anymore and using IO instead? How can I understand which processes cause this?
I am asking this question to stackoverflow users because if I ask it to my hosting provider, I am sure they would ask more money to increase RAM.
Thanks.
This is perfectly normal. When the machine starts up, a large number of services also start up. As they run their startup code, read their configuration, and so on, they dirty some pages of memory. Many of these services will never run again. By writing this data to swap, the operating system accomplishes two things:
First, if it ever does encounter memory pressure, it can discard the pages without having to write them first, since it has already written them. Second, it can discard the pages to make more free memory to enlarge the cache.
The alternative is to keep information that hasn't been touched in days in physical memory. And that just doesn't make sense.
Hi we have a server with 32 cores and 256GB RAM, we are using this with SQL Server 2008 Enterprise on Windows 2008 R2 Enterprise.
Currently windows has allocated automatically a swapfile of 256GB which seems excessive. Is it advisable to hard limit the swapfile to something smaller like 32GB to force it to use the physical RAM?
Is it the swap file or is it the hibernate file?
The answer depends upon the work the machine is expected to do. You might find that Windows doesn't touch the swap file much because you have adequate physical memory available. One approach would be to cut the swap file allocation in half, then use the inbuilt performance monitoring tools to make sure it is still running ok, then after a period of stable running look to half the swap allocation again.
But is it really a problem? With a machine like that you probably have a good chunk of hard drive space available, and i doubt that they would be slow old 5400rpm drives :)
An ideally setup OLTP SQL Server should never need to use the swap file. It depends what you are using this server for.
But unless you are short of disk space, I wouldn't worry too much. 32GB sounds a better size though.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
How to invalidate the file system cache?
I'm writing a disk intensive win32 program. The first time it runs, it runs a lot slower while it scans the user's folders using FindFirstFile()/FindNextFile().
How can I repeat this first time performance without rebooting? Is there any way to force the system to discard everything in its disk cache?
I know that if I were reading a single file, I can disable caching by passing the FILE_FLAG_NO_BUFFERING flag to a call to CreateFile(). But it doesn't seem possible to do this when searching for files.
Have you thought about doing it on a different volume, and dismounting / remounting the volume? That will cause the vast majority of everything to be re-read from disk (though the cache down there won't care).
You need to create enough memory pressure to cause the memory manager and cache manager to discard the previously caches results. For the cache manager, you could try to open a large (I.e. Bigger than physical ram) file with caching enabled and then read it backwards (to avoid any sequential I/o optimizations). The interactions between vm and cache manager are a little more complex and much more dependent on os version.
There are also caches on the controller (possibly, but unlikely) and on the disk drive itself (likely). There are specific IoCtls to flush this cache, but in my experience, disk firmware is untested in this arena.
Check out the Clear function of CacheSet by SysInternals.
You could avoid a physical reboot by using a virtual machine.
I tried all the methods in the answers, including CacheSet, but they would not work for FindFirstFile/FindNextfile(). Here is what worked:
Scanning files over the network. When scanning a shared drive, it seems that windows does not cache the folders, so it is slow every time.
The simplest way to make any algorithm slower is to insert calls to Sleep(). This can reveal lots of problems in multi-threaded code, and that is what I was really trying to do.
Windows Server 2008. How can I quickly use up RAM so to induce GC in my app. If there is a way to do it without needing Visual Studio or installing a language runtime it would be good.
EDIT: I don't want to have to write an app and then copy it over to the server. I'm looking for a way to do it quickly without writing an app that requires an IDE or installation of a runtime/compiler.
Perhaps a powershell or batch script?...
I don't think using up RAM outside your process is going to necessarily trigger GC.
If I understand your question correctly, you have a program Foo.exe that is written in some unknown language, running on some unknown runtime (are you not allowed to post the details for some reason, or do you just not know?), and you want to try to get that program's runtime to trigger a garbage collection. However, you want to do this by using up RAM outside of foo.exe.
You could do this by creating a simple batch file that just started up a hundred copies of IE or Word or whatever program you want. However, I don't think that will do what you want it to do. If your process has already allocated a certain amount of memory, it won't necessarily give that memory up or trigger GC just because other processes are being started. It may page to disk, or may force other programs to page to disk. But not all Garbage Collectors are alike, so we can't really help without more details. I'm pretty sure some VM's never give back memory once they've allocated it, even after GC.
You could run your program inside a virtual machine such as Virtual Box, where you specify the memory ceiling of the guest operating system.
I'm having trouble imagining a scenario where this would be necessary though. Could you provide more information about the problem?
If you are using java you can specify the max amount of memory using Xmx. Search for JVM memory setting