I just want to do rate limiting on a rest api using redi. Could you please suggest me, which datastructure in redis would be appropriate. I just used the RedisTemplate which is not feasible to expire an element, once after updating a key and value.
There are multiple approaches, depending on what exactly you are trying to achieve - from general "ops per second" limiting, to fine grained limits in a lower resolution, like how many posts a specific user can make per day, etc.
One very simple and elegant approach I like is an expiring counter. The technique is simple, and takes advantage of the fact that INCR does not change a key's expiration time in redis. So basically if you want 1000 requests per second on a resource, just create a key with the number 1 (by running INCR) and expire it in a second. Then for each request check if it's reached 1000 and if not increment it. If it has - block the request. When the time window has passed the key will expire automatically and will be recreated on the next request.
In terms of pseudo code, the algorithm is:
def limit(resource_key):
current = GET(resource_key)
if current != NULL and current >= 1000:
return ERROR
else:
value = INCR(resource_key)
IF value == 1:
EXPIRE(value,1)
return OK
Related
How Google Webrisk evaluates the expiryTime and negativeExpiryTime for a particular UpdateAPi request.
Sometimes GWR api return the response with an expirytime of 5-10 minutes and therefore we have to again make an API call to GWR after 5-10 minutes which is pretty soon.
Is there any way to optimise the expiryTime to reduce the cost ?
The expireTime and negativeExpireTime fields indicate until when the hashes must be considered either unsafe or safe respectively in Update API.
To reduce the overall number of hashes.search requests sent to Google using the Update API , clients are required to maintain a local cache. The API establishes two types of caching, positive and negative.
Positive caching (expireTime)
To prevent clients from repeatedly asking about the state of a particular unsafe full hash , each returned ThreatHash contains a positive cache time (defined by the expireTime field). The full hash can be considered unsafe until this time. A positive cache entry should be created or updated for the full hash per the expireTime field.
Negative caching(negativeExpireTime)
To prevent clients from repeatedly asking about the state of a particular safe full hash, the response defines a negative cache duration for the requested prefix (defined by the negativeExpireTime field). All full hashes with the requested prefix are to be considered safe for the requested threat types until this time, except for those returned by the server as unsafe. The hash prefix's negative cache duration should also be created or updated per the response's negativeExpireTime field.
For Example:
Assume a client with an empty cache visits example.com/ and sees that h(example.com/) is in the local database. The client requests the full-length hashes for hash prefix h(example.com/) and receives back the full-length hash H(example.com/) together with a positive cache expireTime of 5 minutes from now and a negative cache expireTime of 1 hour from now.
The positive cache duration of 5 minutes tells the client how long the full-length hash H(example.com/) must be considered unsafe without sending another hashes.search request. After 5 minutes the client must issue another hashes.search request for that prefix h(example.com/) if the client visits example.com/ again. The client should reset the hash prefix's negative cache expireTime per the new response.
Refer to the link for more information about Update API .
I tried googling something similar , but wasn't habel to find something on the topic
I'm just curious, does it matter how big the number of seconds are set in a key impact performance in redis?
For example:
set mykey "foobarValue" EX 100 VS set mykey "foobarValue" EX 2592000
To answer this question, we need to see how Redis works.
Redis maintains tables of a key, value pair with an expiry time, so each entry can be translated to
<Key: <Value, Expiry> >
There can be other metadata associated with this as well. During GET, SET, DEL, EXPIRE etc operations Redis calculates the hash of the given key(s) and tries to perform the operation. Since it's a hash table, it needs to prob during any operation, while probing it may encounter some expired keys. If you have subscribed for "Keyspace notification" then notification would be sent and the given entry is removed/updated based on the operation being performed. It also does rehashing, during rehashing it might find expired keys as well. Redis also runs background tasks to cleanup expire keys, that means if TTL is too small then more keys would be expired, as this process is random, so more event would be generated.
https://github.com/antirez/redis/blob/a92921da135e38eedd89138e15fe9fd1ffdd9b48/src/expire.c#L98
It does have a small performance issue when TTL is small since it needs to free the memory and fix some pointers. But it can so happen that you're running out of memory since expired keys are also present in the database. Similarly, if you use higher expiry time then the given key would present in the system for a longer time, that can create memory issue.
Setting smaller TTL has also more cache miss for the client application, so client will have performance issues as well.
For an API that is delivering time range of performance data down to a granularity of 1 second with primary consumption being live graphs (AJAX) though could be used for alerts as well, would it be better to have the API input take a begin date and end date or a start time and time range?
For example
GET_DATA("beginTime": "2016-01-01T00:00:00Z", "endTime": "2016-01-01T00:05:00Z")
or
GET_DATA("dataTime":"2016-01-01T00:00:00Z","timeWidthSecs":300)
Both versions have advantages.
The first API would be good in the cases where I knew exactly what time range I wanted. An exact time range would be most pertinent if I knew the time range that a performance problem occurred.
The second API is more flexible. Since the main use case is graphing data, the second one would allow me to overload the arguments. For example :
GET_DATA("dataTime":"2016-01-01T00:00:00Z") /* returns all data since time , good for polling and refreshing graph */
GET_DATA("timeWidthSecs":300) /* return most recent 300 secs , good for intial setup of graph */
GET_DATA("dataTime":"2016-01-01T00:00:00Z","timeWidthSecs":300) /* data since time, 300 secs , good for getting historic time range */
GET_DATA("dataTime":"2016-01-01T00:00:00Z","timeWidthSecs":-300) /* data before time, 300 seconds, good for getting historic time range */
Since as a consumer I might be polling , then getting all data since a time, i.e. the last date I have already data for, would be useful. Of course there should be some other construct to stream data for a push model, but for those that wanted a pull model this would be useful.
Also it would be easier to read/debug the code that gives timeWidth in seconds rather than two dates where a human intervention would require the human to do date arithmetic.
Thoughts?
I found 4 "proper" ways to do this:
In the cheat sheet for ActiveRecord users substitutes for ActiveRecord's increment and increment_counter are supposed to be album.values[:column] -= 1 # or += 1 for increment and album.update(:counter_name=>Sequel.+(:counter_name, 1))
In a SO solution update_sql is suggested for the same effect s[:query_volume].update_sql(:queries => Sequel.expr(3) + :queries)
In a random thread I found this one dataset.update_sql(:exp => 'exp + 10'.lit)
In the Sequels API docs for update I found this solution http://sequel.jeremyevans.net/rdoc/classes/Sequel/Dataset.html#method-i-update
yet none of the solutions actually update the value and return the result in a safe, atomic way.
Solutions based on "adding a value and then saving" should, afaik, fail nondeterministically in multiprocessing environments resulting with errors such as:
album's counter is 0
thread A and thread B both fetch album
thread A and thread B both increment the value in the hash/model/etc
thread A and thread B both update the counter to same value
as a result: A and B both set the counter to 1 and work with counter value 1
Sequel.expr and Sequel.+ on the other hand don't actually return a value, but a Sequel::SQL::NumericExpression and (afaik) you have no way of getting it out short of doing another DB roundtrip, which means this can happen:
album's counter is 0
thread A and B both increment the value, value is incremented by 2
thread A and B both fetch the row from the DB
as a result: A and B both set the counter to 2 and work with counter value 2
So, short of writing custom locking code, what's the solution? If there's none, short of writing custom locking code :) what's the best way to do it?
Update 1
I'm generally not happy with answers saying that I want too much of life, as 1 answer suggests :)
The albums are just an example from the docs.
Imagine for example that you have a transaction counter on an e-commerce POS which can accept 2 transactions at the same time on different hosts and to the bank you need to send them with an integer counter unique in 24h (called systan), send 2 trx with same systan and 1 will be declined, or worse, gaps in the counts are alerted (because they hint at "missing transactions") so it's not possible to use the DB's ID value.
A less severe example, but more related to my use case, several file exports get triggered simultaneously in a background worker, every file destination has its own counter. Gaps in the counters are alerted, workers are on different hosts (so mutexes are not useful). And I have a feeling I'll soon be solving the more severe problem anyway.
The DB sequences are no good either because it would mean doing DDL on addition of every terminal, and we're talking 1000s here. Even in my less sever use case DDLing on web portal actions is still a PITA, and might even not work depending on the cacheing scheme below (due to implementation of ActiveRecord and Sequel - and in my case I use both - might require server restart just to register a merchant).
Redis can do this, but it seems insane to add another infrastructure component just for counters when you're sitting on an ACID-compliant database.
If you are using PostgreSQL, you can use UPDATE RETURNING: DB[:table].returning(:counter).update(:counter => Sequel.expr(1) + :counter)
However, without support for UPDATE RETURNING or something similar, there is no way to atomically increment at the same time as return the incremented value.
The answer is - in a multithreaded environment, don't use DB counters. When faced with this dilema:
If I need a unique integer counter, use a threadsafe counter generator that parcels out counters as threads require them. This can be a simple integer or something more complex like a Twitter Snowflake-like generator.
If I need a unique identifier, I use something like a uuid
In your particular situation, where you need a count of albums - is there a reason you need this on the database rather than as a derived field on the model?
Update 1:
Given that you're dealing with something approximating file exports with workers on multiple hosts, you either need to parcel out the ids in advance (i.e. seed a worker with a job and the next available id from a single canonical source) or have the workers call in to a central service which allocates transaction ids on a first come first served basis.
I can't think of another way to do it. I've never worked with a POS system, but the telecoms network provisioning systems I've worked on have generally used a single transaction generator service which namespaced ids as appropriate.
I am wondering how the each command on a Parse Query counts towards the request execution limits. I am building an app that will need to perform a function on many objects (could be more than 1000) in a parse class.
For example (in JavaScript),
var query = new Parse.Query(Parse.User);
query.equalTo('anObjectIWant',true); //there could be more than 1000 objects I want
query.each(function(object){
doSomething(object); //doSomething does NOT involve another Parse request
});
So, will the above code count as 1 request towards my Parse application execution limit (you get 30/second free), or will each object (each recurrence of calling "each") use one request (so 1000 objects would be 1000 requests)?
I have evaluated the resource usage by observing the number of API requests made by query.each() for different result set sizes. The bottom line is that (at the moment of writing) this function is using the default query result count limit of 100. Thus if your query matches up to 100 results it will make 1 API request, 2 API requests for 101-200 and so forth.
This behavior can not be changed by manually increasing the limit to the maximum using query.limit(1000). If you do this you will get an error when you call query.each() afterwards (this is also mentioned in the documentation).
Therefore it has to be considered to manually implement this functionality (e.g., by recursive query.find()) which allows you to set the query limit to 1000 and thus, in the best case, only consumes one-tenth of the API requests query.each() would consume.
This would count as 1 or 2 depending on :
If it is run from cloudcode function =2,when 1 is for cloudcode call + 1 for query. Since queries get their results all at once it is single call.
If this should be place within "beforeSave" functions or similar then only query would be counted, 1 API call.
So you should be pretty fine as long as you don't trigger another parse API for each result.
I would not be surprised if the .each method would query the server each iteration.
You can actually check this using their "control panel", just look at the amount of requests beeing made.
We left Parse after doing some prototyping, one of the reasons was that while using proper and sugested code from the parse website, I managed to create 6500 requests a day beeing the only one using the app.
Using our own API, we are down to not more than 100.