Is it possible to change all the key/value pairs in memcache instances with a command line?
Say, I have 10 memcache servers and they have key value pairs, and they all have the objects with 30 days expiration. But they don't expire at the same time, and I don't want all of them to expire at the same time. I want to change the objects to expire in 10 days. How can I make this change?
Is this even possible?
Can this be done via a commandline? Do I have to write a program for this?
You can accomplish this by touching values periodically. The FAQ describes a way to do this.
However, memcache isn't designed for this. What you're doing seems to be more like a persistent cache scenario. If you love memcache semantics, Membase and MemcacheDB provide solutions that may better fit your needs. There are many different persistent cache systems that do this just as well.
Depending on your specs, sometimes speeding up your data source may deliver better performance than memcache. Modern DMBSs cache heavily with sensible access protocols. This is entirely dependent on what your data sources look like and how much flexibility you have in your system design.
Memcache has a telnet interface. Then you can use FLUSH_ALL or FLUSH_ALL <seconds_to_wait>, if that's what you mean...
Related
I'm trying to understand how rack-attack uses memcached to throttle connections.
As far as I can tell there's no easy way to manage lists in memcached, and no way to search keys by prefix. Yet rack-attack is somehow keeping a list count within cache, but I'm staring at the source code and can't figure out how it works?
https://github.com/kickstarter/rack-attack/blob/master/lib/rack/attack/throttle.rb
https://github.com/kickstarter/rack-attack/blob/master/lib/rack/attack/cache.rb
It's possible to emulate namespacing, tagging and indexing with memcached, which allows you to work around many limitations (in your case you could maintain prefixes as tags). This article has some good ideas, and the memcached docs have some neat tricks too.
I have a specific cache system in Redis.
The content of this system is quite volatile, and values get added and removed all the time. I want to keep the "used" keys in memory as much as possible, while getting the old ones to expire.
Each request can require hundreds of keys from the cache.
I'm aware that I could set a "long enough" expire time, and just dealt with the Cache misses, but I'd like to have as little misses as possible.
Currently I'm doing something like this, when I'm writing / reading to the cache (pseudo code)
# write
write(key, value)
expire(key, ttl)
# read
read(key)
expire(key, ttl)
I can optimise the read by using pipelining.
Now this still seems like it's not the best way of doing it.
Could someone give me a better strategy?
If you can live with the (current) resolution of 10 seconds then the OBJECT IDLETIME command would let you get a better sense of what has not been used for a while (in blocks of 10 seconds)
> SET X 10
OK
> OBJECT IDLETIME X
10
I would create a script (https://redis.io/commands/script-load) that does this atomically and faster directly on the server side and then use it with EvalSha (https://redis.io/commands/evalsha).
This saves the extra round trip on each of the commands.
Alternatively you can implement a similar algorithm to the LRU cache that Redis runs when it's out of space (https://redis.io/topics/lru-cache) - every once in a while get random keys and remove them if they're too old for you, optionally loop until you get a long sequence of new keys.
If what you are trying to achieve is a perfect LRU cache (Least Recently Used), you can tune Redis to behave like this globally, here is a link about Redis as LRU:
http://oldblog.antirez.com/post/redis-as-LRU-cache.html
Note that it is using maxmemory property on redis and the eviction rule is global unless you look at volatile LRU: How to make Redis choose LRU eviction policy for only some of the keys?
You are using a manual solution for eviction with custom expiration / TTL which is the most powerful solution, but maybe you can simplify your configuration and have a better predictable cache in memory size with this solution.
I'm planning the injection of a caching system within my website, will use it in different layers (data, presentation and may be somewhere else). Being my stack LAMP and my infrastructure 100% cloud on AWS, I thought the natural choice would be Amazon Elasticache (a managed installation of memcached). But...
Surprisingly - for me - I discovered memcached completely lacks of dependency management. I don't need "advanced" stuffs like ASP.Net cache SqlDependency or FileDependency, but memcached doesn't offer an easy other-key dependency neither, something pretty useful for building a dependency tree that greatly simplify the invalidation process.
So, as I know memcached is used in many complex systems, am I missing something? Are there usage patterns that make this lack irrelevant?
thanks
UPDATE
as asked, I add some pseudo code to clarify what I mean
dependency = 'ROOT_KEY';
cache:set(dependency, 0, NEVER_EXPIRE);
expire = 600;
cache:set('key1', obj1, expire, dependency);
cache:set('key2', obj2, expire, dependency);
...
cache:set('keyN', objN, expire, dependency);
//later, when I have to invalidate
cache:remove(dependency); //this will cause all keyX to be invalidated too
Based on the example in your question, memcached (and thus Elastic Cache) does not support any sort of key metadata like you are looking for by which you could relate such keys and operate on them as a group.
I suppose if you had only a handful of different "dependencies" you could simply utilize multiple elastic cache instances, which would allow you to invalidate all items within each instance/dependency simultaneously. This of course might end up costing you more in terms of AWS hardware costs then your would like since you can only increment your cache sizes in discrete amounts. This also would eliminate the ability for you to do a cache lookup without knowing the dependency/instance upon which the lookup is to occur.
For what you are trying to do, you might be able to use something like memory tables in MySQL/RDS if you are looking for more of a works-out-of-the-box type of solution. Of course you would not want to use RDS high-availibility features or point-in-time restoration, as these will break, since they require writing to disk. You would basically need to have a standalone RDS instance doing nothing but these memory tables.
It seems none of these options however is really an exact fit for what you are looking to do, so you might need to look into either adjusting your approach (if you want to use basic AWS components), or deploying an alternate caching system on EC2.
Does there exist some sort of persistent key-value like store that allows for quick and easy incrementing, decrementing, and retrieval of integers (and nothing else). I know that I could implement something with a SQL database, but I see two drawbacks to that:
It's heavyweight for the task at hand. All I need is the ability to say "server[key].inc()" or "server[key].dec()"
I need the ability to handle potentially thousands of writes to a single key simultaneously. I don't want to deal with excessive resource contention. Change the value and get out - that's all I need.
I know memcached supports inc/dec, but it's not persistent. My strategy at this point is going to be to use a SQL server behind a queueing system of some sort such that there's only one process updating the database. It just seems... harder than it should be.
Is there something someone can recommend?
Redis is a key-value store that supports several data types. Integer is present, along with incr and decr commands.
Im trying to find a good way to handle memcache keys for storing, retrieving and updating data to/from the cache layer in a more civilized way.
Found this pattern, which looks great, but how do I turn it into a functional part of a PHP application?
The Identity Map pattern: http://martinfowler.com/eaaCatalog/identityMap.html
Thanks!
Update: I have been told about the modified memcache (memcache-tag) that apparently does do a lot of this, but I can't install linux software on my windows development box...
Well, memcache use IS an identity map pattern. You check your cache, then you hit your database (or whatever else you're using). You can go about finding information about the source by storing objects instead of just values, but you'll take a performance hit for that.
You effectively cannot ask the cache what it contains as a list. To mass invalidate, you'll have to keep a list of what you put in and iterate it, or you'll have to iterate every possible key that could fit the pattern of concern. The resource you point out, memcache-tag can simplify this, but it doesn't appear to be maintained inline with the memcache project.
So your options now are iterative deletes, or totally flushing everything that is cached. Thus, I propose a design consideration is the question that you should be asking. In order to get a useful answer for you, I query thus: why do you want to do this?