Nginx: In which order rate limiting and caching are executed? - caching

Question

I want to use nginx for rate limiting and caching.
In which order nginx applies them? In other words, is it limiting only request to the upstream server or all requests (including cache HIT's)?
How can this order be changed? I think it can be changed by having two server contexts. So, for example, on one server performs caching. It has the second server context as upstream. The second one limits requests to the "real" upstream. But that's probably not the most efficient way...

Answer 1

Request Processing
Nginx request processing is is done with a number of different phases, with each phase having one or more handlers. Modules can register to run at a specific phase.
http://www.nginxguts.com/phases/
http://nginx.org/en/docs/dev/development_guide.html#http_phases
Rate Limiting
Rate limiting is applied by the ngx_http_limit_req_module in the pre access phase. From ngx_http_limit_req_module.c:
h = ngx_array_push(&cmcf->phases[NGX_HTTP_PREACCESS_PHASE].handlers);
Caching
Caching is done later, I think in the content phase.
I couldn't quite figure this out from looking at the code or the documentation. But I was able to demonstrate this with a debug build. My configuration had rate limiting 1 request per second, with caching on. See the following excerpts from my log.
Cached Request
...
2020/08/01 11:11:07 [debug] 17498#0: *7 http header done
2020/08/01 11:11:07 [debug] 17498#0: *7 rewrite phase: 0
2020/08/01 11:11:07 [debug] 17498#0: *7 test location: "/"
2020/08/01 11:11:07 [debug] 17498#0: *7 using configuration "=/"
2020/08/01 11:11:07 [debug] 17498#0: *7 http cl:-1 max:1048576
2020/08/01 11:11:07 [debug] 17498#0: *7 rewrite phase: 2
2020/08/01 11:11:07 [debug] 17498#0: *7 post rewrite phase: 3
2020/08/01 11:11:07 [debug] 17498#0: *7 generic phase: 4
2020/08/01 11:11:07 [debug] 17498#0: *7 http script var: ....
2020/08/01 11:11:07 [debug] 17498#0: shmtx lock
2020/08/01 11:11:07 [debug] 17498#0: shmtx unlock
2020/08/01 11:11:07 [debug] 17498#0: *7 limit_req[0]: 0 0.000
2020/08/01 11:11:07 [debug] 17498#0: *7 generic phase: 5
2020/08/01 11:11:07 [debug] 17498#0: *7 access phase: 6
2020/08/01 11:11:07 [debug] 17498#0: *7 access phase: 7
2020/08/01 11:11:07 [debug] 17498#0: *7 post access phase: 8
2020/08/01 11:11:07 [debug] 17498#0: *7 generic phase: 9
2020/08/01 11:11:07 [debug] 17498#0: *7 generic phase: 10
2020/08/01 11:11:07 [debug] 17498#0: *7 http init upstream, client timer: 0
2020/08/01 11:11:07 [debug] 17498#0: *7 http cache key: "http://127.0.0.1:9000"
2020/08/01 11:11:07 [debug] 17498#0: *7 http cache key: "/"
2020/08/01 11:11:07 [debug] 17498#0: *7 add cleanup: 00005609F7C51578
2020/08/01 11:11:07 [debug] 17498#0: shmtx lock
2020/08/01 11:11:07 [debug] 17498#0: shmtx unlock
2020/08/01 11:11:07 [debug] 17498#0: *7 http file cache exists: 0 e:1
2020/08/01 11:11:07 [debug] 17498#0: *7 cache file: "/home/poida/src/nginx-1.15.6/objs/cache/157d4d91f488c05ff417723d74d65b36"
2020/08/01 11:11:07 [debug] 17498#0: *7 add cleanup: 00005609F7C46810
2020/08/01 11:11:07 [debug] 17498#0: *7 http file cache fd: 12
2020/08/01 11:11:07 [debug] 17498#0: *7 read: 12, 00005609F7C46890, 519, 0
2020/08/01 11:11:07 [debug] 17498#0: *7 http upstream cache: 0
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy status 200 "200 OK"
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy header: "Server: SimpleHTTP/0.6 Python/3.8.5"
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy header: "Date: Sat, 01 Aug 2020 01:11:03 GMT"
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy header: "Content-type: text/html; charset=utf-8"
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy header: "Content-Length: 340"
2020/08/01 11:11:07 [debug] 17498#0: *7 http proxy header done
2020/08/01 11:11:07 [debug] 17498#0: *7 http file cache send: /home/poida/src/nginx-1.15.6/objs/cache/157d4d91f488c05ff417723d74d65b36
2020/08/01 11:11:07 [debug] 17498#0: *7 posix_memalign: 00005609F7C46DC0:4096 #16
2020/08/01 11:11:07 [debug] 17498#0: *7 HTTP/1.1 200 OK
...
Rate Limited Request
...
2020/08/01 11:17:04 [debug] 17498#0: *10 http header done
2020/08/01 11:17:04 [debug] 17498#0: *10 rewrite phase: 0
2020/08/01 11:17:04 [debug] 17498#0: *10 test location: "/"
2020/08/01 11:17:04 [debug] 17498#0: *10 using configuration "=/"
2020/08/01 11:17:04 [debug] 17498#0: *10 http cl:-1 max:1048576
2020/08/01 11:17:04 [debug] 17498#0: *10 rewrite phase: 2
2020/08/01 11:17:04 [debug] 17498#0: *10 post rewrite phase: 3
2020/08/01 11:17:04 [debug] 17498#0: *10 generic phase: 4
2020/08/01 11:17:04 [debug] 17498#0: *10 http script var: ....
2020/08/01 11:17:04 [debug] 17498#0: shmtx lock
2020/08/01 11:17:04 [debug] 17498#0: shmtx unlock
2020/08/01 11:17:04 [debug] 17498#0: *10 limit_req[0]: -3 0.707
2020/08/01 11:17:04 [error] 17498#0: *10 limiting requests, excess: 0.707 by zone "mylimit", client: 127.0.0.1, server: , request: "GET / HTTP/1.1", host: "localhost:8080"
2020/08/01 11:17:04 [debug] 17498#0: *10 http finalize request: 503, "/?" a:1, c:1
2020/08/01 11:17:04 [debug] 17498#0: *10 http special response: 503, "/?"
2020/08/01 11:17:04 [debug] 17498#0: *10 http set discard body
2020/08/01 11:17:04 [debug] 17498#0: *10 HTTP/1.1 503 Service Temporarily Unavailable
...
For a rate limited request, processing stops before the server tries to generate the content or check the cache.
TL;DR; Rate limiting is applied first, before caching.

Answer 2

I have faced the same case: if the rate limiter rejects a request then return an already cached response.
In this case you can use error_page directive and "direct" to another location.
proxy_cache_path /var/cache/nginx/html_cache levels=1:2 use_temp_path=off keys_zone=html_cache:128m max_size=5G inactive=7d;
limit_req_zone $binary_remote_addr zone=req_per_ip_limit:64m rate=1r/m;
proxy_cache_key $request_method|$host|$request_uri;
location #html-cache-fallback {
proxy_cache html_cache;
proxy_cache_use_stale error timeout invalid_header http_500 http_502 http_503 http_504 http_403 http_404 http_429;
add_header x-debug-rate-limit REJECTED;
# Set nonexistent host in order to receive error and then return response from cache
proxy_pass http://0.0.0.0:7777;
}
location / {
limit_req zone=req_per_ip_limit;
limit_req_status 429;
# In case rate limit rejects request direct to another location
error_page 429 =200 #html-cache-fallback;
# Cache response
proxy_cache html_cache;
proxy_cache_valid 200 1s;
proxy_cache_use_stale error timeout invalid_header http_500 http_502 http_503 http_504 http_403 http_404 http_429;
proxy_pass https://example.com;
}

Answer 3

If you have network load balancer then you have to implement rate and caching on both server.
Rate limit is like this.
location /login/ {
limit_req zone=mylimit burst=20;
proxy_pass http://my_upstream;
}
The burst parameter defines how many requests a client can make in excess of the rate specified by the zone (with our sample mylimit zone, the rate limit is 10 requests per second, or 1 every 100 milliseconds). A request that arrives sooner than 100 milliseconds after the previous one is put in a queue, and here we are setting the queue size to 20.
That means if 21 requests arrive from a given IP address simultaneously, NGINX forwards the first one to the upstream server group immediately and puts the remaining 20 in the queue. It then forwards a queued request every 100 milliseconds, and returns 503 to the client only if an incoming request makes the number of queued requests go over 20.
Now if you do limit rating on one server only then two subsequent request which goes to different server will have issue.
All caching variable needs to be synced too. you need Redis or persistent storage for caching.
https://www.nginx.com/blog/rate-limiting-nginx/

Answer 4

NGINX
How does Nginx rate limiter works
NGINX rate limiting uses the leaky bucket algorithm, which is widely
used in telecommunications and packet‑switched computer networks to
deal with burstiness when bandwidth is limited. The analogy is with a
bucket where water is poured in at the top and leaks from the bottom;
if the rate at which water is poured in exceeds the rate at which it
leaks, the bucket overflows. In terms of request processing, the water
represents requests from clients, and the bucket represents a queue
where requests wait to be processed according to a first‑in‑first‑out
(FIFO) scheduling algorithm. The leaking water represents requests
exiting the buffer for processing by the server, and the overflow
represents requests that are discarded and never serviced.
Adding snap of config from one of my servers:
The ngx_http_limit_conn_module module is used to limit the number of connections per the defined key, in particular, the number of connections from a single IP address.
Not all connections are counted. A connection is counted only if it has a request being processed by the server and the whole request header has already been read.
So basically you can do two setups for actual and for all other individual virtual servers.
By limiting IP
By limiting Connection
There could be several limit_conn directives. For example, the
following configuration will limit the number of connections to the
server per client IP and, at the same time, the total number of
connections to the virtual server:
Below is the example for the same
limit_conn_zone $binary_remote_addr zone=perip:10m;
limit_conn_zone $server_name zone=perserver:10m;
server {
...
limit_conn perip 10;
limit_conn perserver 100;
}
Together ahead!
Darpan
You can find in detail example and explanation here too