Nginx Proxy to Files on Local Disk or S3 - proxy

Question

So I'm moving my site away from Apache and onto Nginx, and I'm having trouble with this scenario:
User uploads a photo. This photo is resized, and then copied to S3. If there's suitable room on disk (or the file cannot be transferred to S3), a local version is kept.
I want requests for these images (such as http://www.mysite.com/p/1_1.jpg) to first look in the p/ directory. If no local file exists, I want to proxy the request out to S3 and render the image (but not redirect).
In Apache, I did this like so:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^p/([0-9]+_[0-9]+\.jpg)$ http://my_bucket.s3.amazonaws.com/$1 [P,L]
My attempt to replicate this behavior in Nginx is this:
location /p/ {
if (-e $request_filename) {
break;
}
proxy_pass http://my_bucket.s3.amazonaws.com/;
}
What happens is that every request attempts to hit Amazon S3, even if the file exists on disk (and if it doesn't exist on Amazon, I get errors.) If I remove the proxy_pass line, then requests for files on disk DO work.
Any ideas on how to fix this?

Answer 1

Shouldn't this be an example of using try_files?
location /p/ {
try_files $uri #s3;
}
location #s3{
proxy_pass http://my_bucket.s3.amazonaws.com;
}
Make sure there isn't a following slash on the S3 url

Answer 2

You could improve your s3 proxy config like this. Adapted from https://stackoverflow.com/a/44749584:
location /p/ {
try_files $uri #s3;
}
location #s3 {
set $s3_bucket 'your_bucket.s3.amazonaws.com';
set $url_full '$1';
proxy_http_version 1.1;
proxy_set_header Host $s3_bucket;
proxy_set_header Authorization '';
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-meta-server-side-encryption;
proxy_hide_header x-amz-server-side-encryption;
proxy_hide_header Set-Cookie;
proxy_ignore_headers Set-Cookie;
proxy_intercept_errors on;
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;
proxy_pass http://$s3_bucket$url_full;
}

Answer 3

Thanks to keep my coderwall post :) For the caching purpose you can improve it a bit:
http {
proxy_cache_path /tmp/cache levels=1:2 keys_zone=S3_CACHE:10m inactive=24h max_size=500m;
proxy_temp_path /tmp/cache/temp;
server {
location ~* ^/cache/(.*) {
proxy_buffering on;
proxy_hide_header Set-Cookie;
proxy_ignore_headers Set-Cookie;
...
proxy_cache S3_CACHE;
proxy_cache_valid 24h;
proxy_pass http://$s3_bucket/$url_full;
}
}
}
One more recommendation is to extend resolver cache upto 5 min:
resolver 8.8.4.4 8.8.8.8 valid=300s;
resolver_timeout 10s;

Answer 4

break isn't doing quite what you expect nginx will do the last thing you ask of it, which makes sense if you start digging around making modules... but basically protect your proxy_pass with the does-not-exist version
if (-f $request_filename) {
break;
}
if(!-f $request_filename)
proxy_pass http://s3;
}

Answer 5

I ended up solving this by checking to see if the file doesn't exist, and if so, rewriting that request. I then handle the re-written request and do the proxy_pass there, like so:
location /p/ {
if (!-f $request_filename) {
rewrite ^/p/(.*)$ /ps3/$1 last;
break;
}
}
location /ps3/ {
proxy_pass http://my_bucket.s3.amazonaws.com/;
}