Purging nginx cache files does not always work - caching

Question

I run an nginx server + PHP webservices API. I use nginx's fastcgi_cache to cache all GET requests, and when certain resources are updated, I purge one or more related cached resources.
The method I'm using to do this is to calculate the nginx cache file name for each resource I want to purge, and then deleting that file. For the most part, this works well.
However, I've found that sometimes, even after the cache file is deleted, nginx will still return data from cache.
This is not a problem with selecting the correct cache file to delete -- as part of my testing, I've deleted the entire cache directory, and nginx still returns HIT responses
Is anyone aware of why this might be happening? Is it possible that another cache is involved? E.g., could it be that the OS is returning a cached version of the cache file to nginx, so nginx is not aware that it's been deleted?
I'm running this on CentOS, and with this nginx config (minus irrelevant parts):
user nginx;
# Let nginx figure out the best value
worker_processes auto;
events {
worker_connections 10240;
multi_accept on;
use epoll;
}
# Maximum number of open files should be at least worker_connections * 2
worker_rlimit_nofile 40960;
# Enable regex JIT compiler
pcre_jit on;
http {
# TCP optimisation
sendfile on;
tcp_nodelay on;
tcp_nopush on;
# Configure keep alive
keepalive_requests 1000;
keepalive_timeout 120s 120s;
# Configure SPDY
spdy_headers_comp 2;
# Configure global PHP cache
fastcgi_cache_path /var/nginx/cache levels=1:2 keys_zone=xxx:100m inactive=24h;
# Enable open file caching
open_file_cache max=10000 inactive=120s;
open_file_cache_valid 120s;
open_file_cache_min_uses 5;
open_file_cache_errors off;
server {
server_name xxx;
listen 8080;
# Send all dynamic content requests to the main app handler
if (!-f $document_root$uri) {
rewrite ^/(.+) /index.php/$1 last;
rewrite ^/ /index.php last;
}
# Proxy PHP requests to php-fpm
location ~ [^/]\.php(/|$) {
# Enable caching
fastcgi_cache xxx;
# Only cache GET and HEAD responses
fastcgi_cache_methods GET HEAD;
# Caching is off by default, an can only be enabled using Cache-Control response headers
fastcgi_cache_valid 0;
# Allow only one identical request to be forwarded (others will get a stale response)
fastcgi_cache_lock on;
# Define conditions for which stale content will be returned
fastcgi_cache_use_stale error timeout invalid_header updating http_500 http_503;
# Define cache key to uniquely identify cached objects
fastcgi_cache_key "$scheme$request_method$host$request_uri";
# Add a header to response to indicate cache results
add_header X-Cache $upstream_cache_status;
# Configure standard server parameters
fastcgi_split_path_info ^(.+\.php)(/.+)$;
include fastcgi_params;
# php-fpm config
fastcgi_param SCRIPT_URL $fastcgi_path_info;
fastcgi_param PATH_INFO $fastcgi_path_info;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_param REQUEST_SCHEME $scheme;
fastcgi_param REMOTE_USER $remote_user;
# Read buffer sizes
fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
# Keep connection open to enable keep-alive
fastcgi_keep_conn on;
# Proxy to PHP
fastcgi_pass unix:/var/run/php-fpm/fpm.sock;
}
}
}
Now that I look at this, could the open_file_cache parameters be affecting the cache files?
Any ideas?

Answer 1

No, the OS does not cache files.
However, the reason this might be happening is that files are not actually fully deleted until both the link count and the number of processes that have the file open both go down to zero.
The unlink(2) manual page, which documents the system call used by tools like rm, reads as follows:
The unlink() function removes the link named by path from its directory and decrements the link count of the file which was referenced by the link. If that decrement reduces the link count of the file to zero, and no process has the file open, then all resources associated with the file are reclaimed. If one or more processes have the file open when the last link is removed, the link is removed, but the removal of the file is delayed until all references to it have been closed.
Depending on the system, you can actually still recover such open files fully without any data loss, for example, see https://unix.stackexchange.com/questions/61820/how-can-i-access-a-deleted-open-file-on-linux-output-of-a-running-crontab-task.
So, indeed, open_file_cache would effectively preclude your deletion from having any effect within the processes that still have relevant file descriptors in their cache. You may want to use a shorter open_file_cache_valid if urgent purging after deletion is very important to you.