My understanding of consistent hashing is that it is used to minimize rehashing overhead and that request is served to the next nearest worker in the ring to the hash value of the request. So this guarantees same requests to hash to the same worker as long as the worker is up and the hash function doesn't change.
Similar sticky sessions may use cookies to add information to route subsequent requests to the same worker.
Does sticky session use consistent hashing internally or does it use some kind of a map/cache to route sessions to the same worker and both are essentially different entities?
Related
In the context of a highly requested web service written in go language, I am considering to cache some computations. For that, I am thinking to use Redis.
My application is susceptible to receiving an avalanche of requests containing the same payload that triggers a costly computation. So a cache would reward and allow to compute once.
Consider the following figure extracted from here
I use this figure because I think it helps me illustrate the problem. The figure considers the general two cases: the book is in the cache, or this one is not in. However, the figure does not consider the transitory case when a book is being retrieved from the database and other "get-same-book" requests arrive. In this case, I would like to queue the repeated requests temporarily until the book is retrieved. Next, once the book has already arrived, the queued requests are replied with the result which would remain in the cache for fast retrieving of future requests.
So my question asks for approaches for implementing this requirement. I'm considering to use a kind of table on the server (repository) that writes the status of a query database (computing, ready), but this seems a little complicated, because I would need to handle some race conditions.
So I would like to know if anyone knows this pattern or if Redis itself implements it in some way (I have not found it in my consultations, but I suspect that using a Redis lock would be possible)
You can design it as you have described. But there is some things that are important.
Use a unique key
Use an unique key for each book, and if the book is ever changed, that key should also change. This design makes your step (6) save the book in Redis an idempotent operation (you can do it many times with the same result). So you will avoid any race condition with "get-same-book".
Idempotent requests OR asynchronous messages
I would like to queue the repeated requests temporarily until the book is retrieved. Next, once the book has already arrived, the queued requests are replied with the result
I would not recommend to queue requests as you describe it. If the request is a cache-miss, let it retrieve it from the database - but design it idempotent. Alternatively, you should handle all requests as asynchronous, and use a message queue e.g. nats, RabbitMQ or something, but the complexity grows with that solution.
Serializing requests
My problem is that while that second of computation where the result is not still gotten, too many repeated requests can arrive and due to the cost I need to avoid to repeat their computations. I need to find a way of retaining them while the result of the first request arrives.
It sounds like you want to have your computations serialized instead of doing them concurrently because you want to avoid doing the same computation twice. To solve this, you should let the requests initialize the computation, e.g. by putting the input on a queue and then do the computation in a serial order (but still possibly concurrently if they have a different key) and finally notify the client, or if the client is subscribing for updates (a better solution).
Redis do have support for PubSub but it depends on what requirements you have on the clients. I would recommend a solution without locks, for scalability.
I have a single-server system that runs all kinds of computations on user data, accessible via REST API. The computations require that large chunks of the user data are in memory during the computation. To do this efficiently, the system includes an in-memory cache, so that multiple requests on the same data chunks will not need to re-read the chunks from storage.
I'm now trying to scale the system out, since one large server is not enough, and I also want to achieve active/active high availability. I'm looking for the best practice to load balance between the multiple servers, while maximizing the efficiency of the local cache already implemented.
Each REST call includes a parameter that identifies which chunk of data should be accessed. I'm looking for a way to tell a load balancer to route the request to a server that has that chunk in cache, if such a server exists - otherwise just use a regular algorithm like round robin (and update the routing table such that the next requests for the same chunk will be routed to the selected server).
A bit more input to consider:
The number of data chunks is in the thousands, potentially tens of thousands. The number of servers is in the low dozens.
I'd rather not move to a centralized cache on another server, e.g. Redis. I have a lot of spare memory on the existing machines that I'd like to utilize since the computations are mostly CPU-bound. Also, I'd prefer not re-implement another custom caching layer.
My servers are on AWS so a way to implement this in ELB is fine with me, but open to other cloud-agnostic solutions. I could in theory implement a system that updates rules on an AWS application load balancer, but it could potentially grow to thousands of rules (one per chunk) and I'm not sure that will be efficient.
Since requests using the same data chunk can come from multiple sources, session-based stickiness is not enough. Some of these operations are write operations, and I'd really not want to deal with cross-server synchronization. All the operations on a single chunk should be routed to the single server that has that chunk in memory.
Any ideas are welcome! Thanks!
How to mantain session b/w 2 different instances of tomcat.
not asking about sticky session
do not want to use any token machenisum like fedrated login/o-outh
Well, the formal answer to this should be:
Use the session replication mechanism. For example, for Tomcat 7 is described here.
So if you know that the application is very small (say, less than 100 users, although it really depends on the data you're planning to store there) you can safely use it, and stop reading my answer here :)
However, in my opinion, small applications tend to grow, the number or users tends to increase, so the best would be not to maintain the session at all :). Because if you need session, this means that you store on server the information about the client. So when the number of client grows much, you won't be able to maintain that.
An alternative is:
Store the session information in some very-fast storage, for example Redis
Maintain some identifier of the user (the chances are that you already have one) and just query redis for the data. Redis has TTLs, so the data can be removed from Redis automatically.
The advantage is that the solution like this is much more scalable, so it can easily handle literally millions of records, while maintaining this at the level of JVM will make it impossible to scale.
Is it possible to persist only certain keys to disk using Redis? Is the best solution for this as of right now to run separate Redis servers where one server can have throw away caches and the other one has more important data that we need to flush to disk periodically (such as counters to visits on a web page)
You can set expirations on a subset of your keys. They will be persisted to disk, but only until they expire. This may be sufficient for your use case.
You can then use the redis maxmemory and maxmemory-policy configuration options to cap memory usage and tell redis what to do when it hits the max memory. If you use the volatile-lru or volatile-ttl options Redis will discard only those keys that have an expiration when it runs out of memory, throwing out either the Least Recently Used or the one with the nearest expiration (Time To Live), respectively.
However, as stated, these values are still put to disk until expiration. If you really need to avoid this then your assumption is correct and another server looks to be the only option.
When you have peaks of 600 requests/second, then the memcache flushing an item due to the TTL expiring has some pretty negative effects. At almost the same time, 200 threads/processes find the cache empty and fire of a DB request to fill it up again
What is the best practice to deal with these situations?
p.s. what is the term for this situation? (gives me chance to get better google results on the topic)
If you have memcached objects which will be needed on a large number of requests (which you imply is the case), then I would look into having a separate process or cron job that regularly calculated and refreshed these objects. That way it should never go TTL. It's a common trade-off: you add a little unnecessary load during low traffic time to help reduce the load during peaking (the time you probably care the most about).
I found out this is referred to as "stampeding herd" by the memcached folks, and they discuss it here: http://code.google.com/p/memcached/wiki/NewProgrammingTricks#Avoiding_stampeding_herd
My next suggestion was actually going to be using soft cache limits as discussed in the link above.
If your object is expiring because you've set an expiry and it's gone past date, there is nothing you can do but increase the expiry time.
If you are worried about stale data, a few techniques exist you could consider:
Consider making the cache the authoritative source for whatever data you are looking at, and make a thread whose job is to keep it fresh. This will make the other threads block on refilling the cache, so it may only make sense if you can
Rather than setting a TTL on the data, change whatever process updates the data to update the cache. One technique I use for frequently changing data is to do this probabilistically -- 10% of the time data is written, it is updated. You can tune this for whatever is sensible, depending on how expensive the DB query is and how severe the impact of stale data.