Nginx and/or php5-fpm remembers symlinked root directory - caching

Question

My nginx site root points to a symlink. If I alter the symlink (aka deploy a new version of the website) the old version of the php script keeps showing up.
That smells like cache, or a bug.
First it looked like Nginx was caching the symlinked dir, but reloading/restarting/killing and starting nginx didn't fix it, so I restarted php5-fpm - this fix my issue.
But I dont want to restart nginx and/or php5-fpm after a deploy - I want to know why there is such a cache (or bug), and why it didn't work properly.
Usefull information:
OS: Ubuntu 13.10 (GNU/Linux 3.8.0-19-generic x86_64)
Nginx: via ppa:nginx/stable
PHP: via ppa:ondrej/php5 (php5-fpm)
Nginx site config:
root /home/rob/sandbox/deploy/public/;
index index.php index.html index.htm;
location / {
try_files $uri $uri/ /index.php?$args;
}
location ~ \.php$ {
try_files $uri =404;
fastcgi_split_path_info ^(.+\.php)(/.+)$;
include fastcgi_params;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_pass php;
}
Nginx server config (partly, rest is default):
http {
sendfile off;
upstream php {
server unix:/var/run/php5-fpm.sock;
}
}
Tree for /home/rob/sandbox:
├── deploy -> web2
├── web1
│ └── public
│ └── index.php (echo ONE)
└── web2
└── public
└── index.php (echo TWO)
request: http://localhost/index.php
expected response: TWO
actual response: ONE
Part of the output from realpath_cache_get()
[/home/rob/sandbox/deploy/public/index.php] => Array (
[key] => 1.4538996210143E+19
[is_dir] =>
[realpath] => /home/rob/sandbox/web2/public/index.php
[expires] => 1383730041
)
So this means deploy/public/index.php is properly linked to web2/public/index.php, right?
Well, even with the correct paths in the realpath_cache list, the respone still is ONE.
After rm deploy and ln -s web2 deploy Nginx was restarted, no effect.
Restarting php5-fpm after this gives the expected response of 'TWO'.
It's good to know that beside the index.php files, I did some test with .css and .js files.
After removing and recreating the symlink from/to web1 and web2, nginx will respond with the correct contents of the files.
What did I miss, what am I not seeing?

Answer 1

Configure your nginx using $realpath_root. It should help.
fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name;
fastcgi_param DOCUMENT_ROOT $realpath_root;
Kudos go to Vitaly Chirkov (https://stackoverflow.com/a/23904770/219272).

Answer 2

Once I altered the realpath_cache_ttl to '2' (That should fix it) the incorrect content was still showing.
After some digging in the loaded mods for php-fpm, I discovered that opcache was up and running. Disabling that will clear the cached realpath's when the ttl is over.
I don't wanna lower the realpath cache ttl to much,so I will settle in with a reload of php-fpm, since it is graceful.
I hope this thread and my answers will help others ;)

Answer 3

I had exactly same problem and none of the tricks did help. Beside all tricks listed on this page I ensured opcache is disabled then nginx cache was also disabled. I set
sendfile off;
It was described somewhere on stackoverflow.
I started to strace the php and nginx and it turned out that some libraries are read new location but also some were read from OLD location that symlink does not point to anymore.
On top of that updating the php script inside OLD directory was always showing properly - so that did not look like cache to me.
To make it more confusing command line worked fine and followed the symlink to NEW location.
I was pulling hair from my head over this!
It turned out that responsible for all this was the composer cache - it was caching some libraries.
I know not everyone uses it, but if you do and have similar problem it is worth checking.
I have vendor at the same level as the deployment scripts and I assume composer cache was causing a lot of confusion which location should be used.
After cleaning it up with
composer clear-cache
System started to behave as expected.
I hope it will help someone.