I fired up a Zilla instance of heroku postgres which is advertised as having 17GB of memory cache.
When I run show all; I see:
effective_cache_size | 12240000kB
Does this mean the cache is 12GB and not 17GB? Or am I missing something? Queries run much slower when my dataset goes above the 12GB point.
There is a limit on the available memory on the underlying hardware (17G for a zilla). This amount of memory cannot be used entirely for the "hot dataset" cache, however. Many other aspects of normal postgres operations also require memory, as you may imagine. Some of that includes establishing a connection (which spawns a backend), queries requiring joins, queries requiring sorts, or aggregates like count, sum, max, etc. Additionally, processes such as autovacuum also use part of that available memory.
Related
There is something i did not really understand with Amazon RDS (the PostrgreSQL version). Some queries takes a lot of time to show their results. I have set all relevant indexes (as shown with EXPLAIN). So I think it's not due to my schema design.
I do not use a big machine (m3.xlarge) as bigger ones are too much expensive. My database size is about 300GB.
It seems that Postgres does not use all the available memory (only ~5GB, the "Freeable memory" report of the console shows that there are always ~10GB freeable...). I try to tune my "parameter group" as proposed by tune-your-postgres-rds-instance, especially set EFFECTIVE_CACHE_SIZE to 70%. But it does not change anything.
I'm probably wrong somewhere... Any idea ?
To make more memory available to your queries you would tune your work_mem.
There are implications to doing that since that's memory per backend.
effective_cache_size actually doesn't deal with memory at all. It's an optimizer parameter.
"Freeable memory" is a good thing - it means that the memory is currently used (most likely) by postgres in the operating system cache.
You can increase your shared_buffers to allow postgres to use more of it's own memory for caching, but there are limits to it's effectiveness that mean you don't usually want to use more than 25% of available memory to this.
Suppose you had a server with 24G RAM at your disposal, how much memory would you allocate to (Tomcat to run) eXist?
I'm setting up our new webserver, with an Intel Xeon E5649 (2.53GHz) processor, running Ubuntu 12.04 64-bit. eXist is running as a webapp inside Tomcat, and the db is only used for querying 'stable' collections --that is, no updates are being executed to the resources inside eXist.
I've been experimenting with different heap sizes (via -Xms and -Xmx settings when starting the Tomcat process), and so far haven't noticed much difference in response time for queries against eXist. In other words, it doesn't seem to matter much whether the JVM is allocated 4G or 16G. I have also upped the #cachesize and #collectionCache in eXist's WEB-INF/conf.xml file to e.g. 8192M, but this doesn't seem to have much effect. I suppose these settings /do/ have an influence when eXist is running inside Tomcat?
I know each situation is different (and I know there's a Tomcat server involved), but are there some rules of thumb for eXist performance w.r.t. the memory it is allocated? I'd like to get at a sensible memory configuration for a setup with a larger amount of RAM available.
This question was asked and answered on the exist-open mailing list. The answer from wolfgang#exist-db.org was:
Giving more memory to eXist will not necessarily improve response times. "Bad"
queries may consume lots of RAM, but the better your queries are optimized, the
less RAM they need: most of the heavy processing will be done using index
lookups and the optimizer will try to reduce the size of the node sets to be
passed around. Caching memory thus has to be large enough to hold the most
relevant index pages. If this is already the case, increasing the caching space
will not improve performance anymore. On the other hand, a too small cacheSize
of collectionCache will result in a recognizable bottleneck. For example, a
batch upload of resources or creating a backup can take several hours (instead
of e.g. minutes) if #collectionCache is too small.
If most of your queries are optimized to use indexes, 8gb RAM for eXist does
usually give you enough room to handle the occasional high load. Ideally you
could run some load tests to see what the maximum memory use actually is. For
#cacheSize, I rarely have to go beyond 512m. The setting for #collectionCache
depends on the number of collections and documents in the database. If you have
tens or hundreds of thousands of collections, you may have to increase it up to
768m or more. As I said above, you will recognize a sudden breakdown in
performance during uploads or backups if the collectionCache becomes too small.
So to summarize, a reasonable setting for me would be: -Xmx8192m,
#cacheSize="512m", #collectionCache="768m". If you can afford giving 16G main
memory it certainly won’t hurt. Also, if you are using the lucene index or the
new range index, you should consider increasing the #buffer setting in the
corresponding index module configurations in conf.xml as well:
<module id="lucene-index" buffer="256" class="org.exist.indexing.lucene.LuceneIndex" />
<module id="range-index" buffer="256" class="org.exist.indexing.range.RangeIndex"/>
It's a frequently asked question, I know. But I tried every solution suggested (apc.stat=0, increasing shared memory, etc) without benefits.
Here's the screen with stats (you can see nginx and php5-fpm) and the parameters set in apc.ini:
APC is used for system and user-cache entries on multiple sites (8-9 WordPress sites and one with MediaWiki and SMF).
What would you suggest?
Each wordpress site is is going to cache a healthy amount in the user cache. I've looked at this myself in depth and have found the best 'guesstimate' is that if you use user cache in APC keep the fragmentation under 10%. This can sometimes mean you need to try and reserve upwards of 10x the amount of memory you intend to actually use for caching to avoid fragmentation. Start where you are and keep increasing memory allocated until fragmentation stays below 10% after running a while.
BTW: wordpress pages being cached are huge, so you'll probably need a lot of memory to avoid fragmentation.
Why 10% fragmentation? It's a bit of a black art, but I observed this is where performance start to noticeabily decline. I haven't found any good benchmarks (or run my own controlled tests) however.
This 10x amount to get this my seem insane, but root cause is APC has no way to defragment other than a restart (which completely dumps the cache). Having a slab of 1G of memory when you only plan to use 100-200m gives a lot of space to fill without having to look for 'holes' to put stuff. Think about bad old FAT16 or FAT32 disk performance with Windows 98--constant need to manually defrag as the disk gets over 50% full.
If you can't afford the extra memory to spare, you might want to look at either memcached or plain old file caching for your user cache.
I don't think it's clear to me yet, is it faster to read things from a file or from memcached? Why?
Memcached is faster, but the memory is limited. HDD is huge, but I/O is slow compared to memory. You should put the hottest things to memcached, and all the others can go to cache files.
(Or man up and invest some money into more memory like these guys :)
For some benchmarks see: Cache Performance Comparison (File, Memcached, Query Cache, APC)
In theory:
Read 1 MB sequentially from memory 250,000 ns
Disk seek 10,000,000 ns
http://www.cs.cornell.edu/projects/ladis2009/talks/dean-keynote-ladis2009.pdf
There are quite a few different aspects that might favour one or the other:
Do you need/want to share this data between multiple servers? The filesystem is local, memcached is accessed over a network.
How large are the items your caching? The filesystem is likely to be better for large objects.
How many memcached requests might be there per page? TCP connections and tear-downs might take up more time than a simple filesystem stat() on the local machine.
I would suggest you look at your use case and do some profiling of both approaches. If you can get away with using the filesystem then I would. Adding in memcached adds in another layer of complexity and potential points of failure (memcached client/server).
For what it's worth the other comments about disk vs memory performance might well be academic as if the filesystem data is being accessed regularly then it'll likely be sitting in OS or disk cache memory anyway.
"Faster" can not be used without context.
For example, accessing data in memcached on remote server can be "slower" due to network latency. In the other hand, reading data from remote server memory via 10Gb network can be "faster" than reading same data from local disk.
The main difference between caching on the filesystem and using memcached is that memcached is a complete caching solution. So there is LRU lists, expiration concept (data freshness), some high-level operations, like cas/inc/dec/append/prepend/replace.
Memcached is easy to deploy and monitor (how can we distinguish "cache" workload on filesystem from, let's say kernel? Can we calculate total amount of cached data? Data distribution? Capacity planning? And so on).
There are also some hybrid systems, like cachelot
Basically, it's memcached that can be embedded right into the application, so the cache would be accessible without any syscalls or network IO.
In fact, it is not as simple as that reading from memory is much faster than reading from HDD. As you known, Memcached is based on tcp connection, if you make connection each time you want to get sth or set sth to memcached server(that is most of programmers do), it proberly performs poorly than using file cache. You should use static Memcached object, and reuse the object. Secondly, the modern OS's will cached files that are frequently used, that makes file caches might be faster than memcaches which are actualy TCP connections.
Cache Type | Cache Gets/sec
Array Cache | 365000
APC Cache | 98000
File Cache | 27000
Memcached Cache (TCP/IP) | 12200
MySQL Query Cache (TCP/IP) | 9900
MySQL Query Cache (Unix Socket) | 13500
Selecting from table (TCP/IP) | 5100
Selecting from table (Unix Socket) | 7400
Source:
https://surniaulula.com/os/unix/memcached-vs-disk-cache/
Source of my source :)
https://www.percona.com/blog/2006/08/09/cache-performance-comparison/
You're being awefully vauge on the details. And I believe the answer your looking for depends on the situtation. To my knowledge very few things tend to be better than the other all the time.
Obviously it woudln't be faster to read things of the file system (assuming that it's a harddrive). Even a SDD will be noticably slower than in-memory reads. And the reason for that is that HDD/FileSystem is built for capacity not speed, while DDR memory is particulary fast for that reason.
Good caching means to keep frequently accessed parts in memory and the not so frequently accessed things on disk (persistent storage). That way the normal case would be vastly improved by your caching implementation. That's your goal. Make sure you have a good understanding of your ideal caching policy. That will require extensive benchmarking and testing.
It depends if the cache is stored locally. Memcache can store data across a network, which isn't necessarily faster than a local disk.
If that file is stored in disk, and it is accessed frequently, there will be a high probability of finding it in the RAM (as a recently accessed file) or I am missing something here?
Yes the first read will be from the disk which is awefully slow, but what about the subsequent reads (assuming that file is hot and its getting lots of reads) this should be even faster than memcached as this is a pure RAM read
In the app I'm writing, I use a lot of in memory containers (C++ std containers but I don't think that's relevant).
During one "task" of my app, in a heavy usage edge-case the private bytes memory usage hits 1GB.
Just as a bit of context, this task is a user initiated task involving 100,000s files. It's likely that the user will kick this off and then leave the machine running.
(And no I don't do anything dumb like load files into memory - this footprint is all metadata related to the task in progress).
For most users the memory usage during this task is negligable - it's just the 1% of users who want to do 500,000 "things" insted of 5000 "things".
I was about to embark on a process to move a lot of this in-memory stuff to disk somehow, e.g. scratch file, embedded DB.
But then I thought - "hang on a minute. All of these solutions are essentially caching memory to disk. Isn't that what Virtual Memory is for?".
I'm not interested in persisting this data - it's purely scratch/temporary stuff I need access to while the task is running.
So my question is, what should I do?
I don't want to do a major refactor for that 1%, but I want to know the impact of running an app with that high a memory footprint.
Am I right in saying that I probably wouldn't be able to do much better than the Windows VM manager anyway?
Under what conditions does this become harmful? OK so yes, if I used up all the real memory then it'd be thrashing to reload pages. But wouldn't I have that anyway in the case if e.g. an embedded database?
Cheers,
John
Yes, the memory manager will do the job for you. Not without side-effects though, it is going to evict pages from RAM that other processes have mapped and give them to you. Those other processes are going to get slowed down by this, they'll be hit with a page fault when they access such a swapped-out page.
The balancing act here is whether your app is "important" enough to justify those other processes from getting short shrift. Usually that's Yes on a work station, a resounding No on a server.