I'm trying to get my head around caching images for my open source image hosting serivce PictShare.
Pictshare has a smart query system where an uploaded image can be in a "virtual subdirectory" that changes the image. For example this is the link to the uploaded stackoverflow logo: https://www.pictshare.net/6cb55fe938.png
I can resize it to 300 width by adding /300/ to the URL before the image name: https://www.pictshare.net/300/6cb55fe938.png
Since I'm dealing with a lot of traffic lately I want my nginx proxy to be able to cache all images in from all virtual sub folders but it's not working. I've read many articles and many stackoverflow posts but no solution worked for me.
So far this my productive vhost file
proxy_cache_path /etc/nginx/cache/pictshare levels=1:2 keys_zone=pictshare:50m max_size=1000m inactive=30d;
proxy_temp_path /etc/nginx/tmp 1 2;
proxy_cache_key "$scheme$request_method$host$request_uri";
proxy_ignore_headers "Set-Cookie";
proxy_hide_header "Set-Cookie";
proxy_buffering on;
server {
...
location / {
proxy_pass http://otherserver/pictshare/;
include /etc/nginx/proxy_params;
location ~* \.(?:jpg|jpeg|gif|png|ico)$ {
expires max;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
proxy_cache_valid 200 301 302 1y;
proxy_cache pictshare;
proxy_pass http://otherserver/pictshare$request_uri;
}
}
}
The problem is that no files are cached and I see every image request on the proxy destination.
The only way I got it to work was by adding a special location to the host file that has caching explicitly enabled:
location /cached {
proxy_cache_valid 200 1y;
proxy_cache pictshare;
expires 1y;
proxy_cache_use_stale error timeout http_500 http_502 http_503 http_504;
include /etc/nginx/proxy_params;
proxy_pass http://otherserver/pictshare/thumbs/;
proxy_cache_use_stale error timeout invalid_header updating http_500 http_502 http_503 http_504;
}
The obvious problem with this solution is that the images are only cached when the request starts with /cached eg: https://www.pictshare.net/cached/6cb55fe938.png
Adding the caching commands to the root directory is no option for me since I don't want the forms and pages to be cached, just the images
Where is my mistake?
proxy_cache_path /etc/nginx/cache/pictshare levels=1:2 keys_zone=my_cache:50m max_size=3g inactive=180m;
proxy_temp_path /etc/nginx/tmp 1 2;
proxy_cache_key "$scheme$request_method$host$request_uri";
location ~* ^.+\.(jpe?g|gif|png|ico|pdf)$ {
access_log off;
include /etc/nginx/proxy.conf;
proxy_pass http://backend;
proxy_cache pictshare;
proxy_cache_valid any 12h;
add_header X-Proxy-Cache $upstream_cache_status;
root /var/www/public_html/cached; }
location / {
include /etc/nginx/proxy.conf;
proxy_pass http://backend;
root /var/www/public_html;
}
nginx first searches for the most specific prefix location given by literal strings regardless of the listed order. In the configuration above the only prefix location is “/” and since it matches any request it will be used as a last resort. Then nginx checks locations given by regular expression in the order listed in the configuration file. The first matching expression stops the search and nginx will use this location. If no regular expression matches a request, then nginx uses the most specific prefix location found earlier.
http://nginx.org/en/docs/http/request_processing.html
Related
I am trying to get nginx running as a cache proxy. The server is up and running properly - except it seems its not caching properly.
According to a scan with pingdom:
The following cacheable resources have a short freshness lifetime. Specify
an expiration at least one week in the future for the following resources:
/layout/theme1/css/bootstrap-responsive.css
/layout/theme1/css/bootstrap.min.css
/layout/theme1/css/font-awesome.css
/layout/theme1/css/pages/dashboard.css
/layout/theme1/css/style.css
/layout/theme1/css/theme.css
/layout/theme1/img/logo.png
/layout/theme1/js/bootstrap.js
/layout/theme1/js/chart.min.js
/layout/theme1/js/excanvas.min.js
/layout/theme1/js/jquery.js
My nginx proxy_cache settings are as follows:
Proxy Cache Path Settings:
proxy_cache_path /var/cache/nginx/marketers.coop levels=1:2 keys_zone=marketers.coop:10m inactive=525600m max_size=5120m;
location settings:
location / {
proxy_pass http://192.227.210.138:8080;
proxy_cache cache;
proxy_cache_valid 30d;
proxy_cache_valid 404 30d;
proxy_no_cache $no_cache;
proxy_cache_bypass $no_cache;
proxy_cache_bypass $cookie_session $http_x_update;
location ~* ^.+\.(jpg|jpeg|gif|png|ico|svg|css|zip|tgz|gz|rar|bz2|doc|xls|exe|pdf|ppt|txt|odt|ods|odp|odf|tar|wav|bmp|rtf|js|mp3|avi|mpeg|flv|html|htm)$ {
proxy_pass http://192.227.210.138:8080;
proxy_cache cache;
proxy_cache_valid 30d;
proxy_cache_valid 404 30d;
root /home/admin/web/marketers.coop/public_html;
access_log /var/log/apache2/domains/marketers.coop.log combined;
access_log /var/log/apache2/domains/marketers.coop.bytes bytes;
expires 365d;
try_files $uri #fallback;
}
}
As can bee seen both css and js are set to be cached, yet neither the expire of 365 days nor the proxy cache valid for 30 days seems to be having any effect. What am I doing wrong?
I'm using "Filling the Cache Slice-by-Slice" from this article https://www.nginx.com/blog/nginx-caching-guide/
proxy_cache_path /tmp/mycache keys_zone=mycache:10m;
server {
listen 80;
proxy_cache mycache;
slice 1m;
proxy_cache_key $host$uri$is_args$args$slice_range;
proxy_set_header Range $slice_range;
proxy_http_version 1.1;
proxy_cache_valid 200 206 1h;
location / {
proxy_pass http://origin:80;
}
}
nginx is caching video from server, but browser is not caching video. Help me please.
I'm solved problem. I'm added ETag and Last-Modified on backend.
I'm having issues getting NGINX to cache thumbnails that I'm pulling from Dropbox using the proxy_pass command. On the same server that NGINX is running I run the following command multiple times
wget --server-response --spider http://localhost:8181/1/thumbnails/auto/test.jpg?access_token=123
and get the exact same response with X-Cache: MISS every time
HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Wed, 25 Mar 2015 20:05:36 GMT
Content-Type: image/jpeg
Content-Length: 1691
Connection: keep-alive
pragma: no-cache
cache-control: no-cache
X-Robots-Tag: noindex, nofollow, noimageindex
X-Cache: MISS
Here's my meat of my nginx.conf file .. any ideas on what I'm doing wrong here?
## Proxy Server Caching
proxy_cache_path /data/nginx/cache keys_zone=STATIC:10m max_size=1g;
## Proxy Server Setting
server {
listen *:8181;
proxy_cache STATIC;
proxy_cache_key "$request_uri";
proxy_cache_use_stale error timeout invalid_header updating
http_500 http_502 http_503 http_504;
location ~ ^/(.*) {
set $dropbox_api 'api-content.dropbox.com';
set $url '$1';
resolver 8.8.8.8;
proxy_set_header Host $dropbox_api;
proxy_cache STATIC;
proxy_cache_key "$request_uri";
proxy_cache_use_stale error timeout invalid_header updating
http_500 http_502 http_503 http_504;
add_header X-Cache $upstream_cache_status;
proxy_pass https://$dropbox_api/$url$is_args$args;
}
##Error Handling
error_page 500 502 503 504 404 /error/;
location = /error/ {
default_type text/html;
}
}
Turns out that thumbnail requests returned from Dropbox include the header
Cache-Control: no-cache
and Nginx will adhere to these headers unless they are explicitly ignored which can be done by simply using the following config line that will ignore any caching control.
proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
We also had issues placing the "proxy_ignore_headers" option in different areas within the nginx.conf file. Finally after much playing around we got it to work by explicitly setting it in the "location" block. The full snippet of the config file can be found below
## Proxy Server Caching
proxy_cache_path /data/nginx/cache levels=1:2 keys_zone=STATIC:50m inactive=2h max_size=2g;
## Proxy Server Setting
server {
listen *:8181;
location ~ ^/(.*) {
set $dropbox_api 'api-content.dropbox.com';
set $url '$1';
resolver 8.8.8.8;
proxy_set_header Host $dropbox_api;
proxy_hide_header x-dropbox-thumbcachehit;
proxy_hide_header x-dropbox-metadata;
proxy_hide_header x-server-response-time;
proxy_hide_header x-dropbox-request-id;
proxy_hide_header cache-control;
proxy_hide_header expires;
add_header cache-control "private";
add_header x-cache $upstream_cache_status; # HIT / MISS / BYPASS / EXPIRED
proxy_cache STATIC;
proxy_cache_valid 200 1d;
proxy_cache_use_stale error timeout invalid_header updating
http_500 http_502 http_503 http_504;
proxy_ignore_headers X-Accel-Expires Expires Cache-Control;
proxy_pass https://$dropbox_api/$url$is_args$args;
}
}
In order to cache the proxy response the request between Nginx and origin should be cookie-less:
proxy_hide_header Set-Cookie;
proxy_ignore_headers Set-Cookie;
See full configuration with invalidation methods: https://gist.github.com/mikhailov/9639593
If above answers didn't solved your issue, try this:
proxy_cache_valid 200 2d;
(or whatever amount of time and whatever response code you want)
add this where you are using or activating your proxy_cache <keys_zone_name>.
Apparently for me as soon as I remove proxy_cache_valid parameter caching status doesn't show up. Also documentation doesn't says that this is required field. Let me know if this works for you. So we might update documentation.
I expect expect proxy_cache get started page should show that you need 3 at least these parameters to get started: proxy_cache_path, proxy_cache and proxy_cache_valid
Well, i have my site.conf file like this:
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_cache_key "$scheme$request_method$host$request_uri";
proxy_cache_path /etc/nginx/cache/pag levels=1:2 keys_zone=APP:100m inactive=1m;
proxy_temp_path /etc/nginx/cache/tmp;
add_header X-Cache $upstream_cache_status;
server {
listen 80;
root /etc/nginx/html;
index index.html index.htm;
server_name www.example.com;
error_page 404 /404.html;
location /404.html {
internal;
}
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_cache APP;
proxy_cache_valid 200 1m;
proxy_cache_methods POST;
expires 1m;
}
}
With this configuration, everything (including POST request methods) is cached for 1 min, OK.
What i need? I need that only this pages can be cached:
1) www.example.com
2) www.example.com/index.html
3) www.example.com/test/page.html
4) www.example.com/test/text.txt (this is a file requested by POST thru page.html, and i need it cached also)
5) www.example.com/test/page2.php?var1=val1&var2=val2 (val1 and val2 are dynamics)
My question is: What i have to put in location / to match the 1-5 items? Like this:
location (1-5 items match) {
proxy_pass http://127.0.0.1:8080/;
proxy_cache APP;
proxy_cache_valid 200 1m;
proxy_cache_methods POST;
expires 1m;
}
Other pages (not cached) will be automatically redirected to 127.0.0.1:8080. I know this can be do like this:
location / {
proxy_pass http://127.0.0.1:8080/;
}
NOTE 1: Other PHP pages receive POST|GET request methods, but i don't need it in cache, only aboves.
NOTE: 2 127.0.0.1:8080 is an apache server that runs PHP, so i can request PHP pages.
Since apache runs on the same host, simply serve the html files you do not want cached through nginx. As for the php pages, send the correct expiration headers in your application and everything will work correctly.
I tried to setup at least 2 servers with nginx (origin + edge). both compiled with the mp4-module. The origin holds all my mp4-files. Edge is configured with all the caching-stuff (see below) that work as expected, each mp4-file request a second time is served by the edge-cache without origin traffic.
But I want to be able to seek in the file. The functionality comes from the mp4-module. Just append the query-param "?start=120" tells nginx to serve the mp4-content starting with timestamp 120sec. This works fine with origin directly requested. But as soon as i enable mp4-module in the caching-location of the nginx, the request will be 404.
nginx.conf # origin:
server {
listen 80;
server_name localhost;
root /usr/share/nginx/www;
location ~ \.mp4$ {
mp4;
expires max;
}
}
nginx.conf # edge:
proxy_cache_path /usr/share/nginx/cache levels=2:2 keys_zone=icdn_cache:10m inactive=7d max_size=2g;
proxy_temp_path /usr/share/nginx/temp;
proxy_ignore_headers X-Accel-Expires Cache-Control Set-Cookie;
log_format cache '[$time_local] Cache: $upstream_cache_status $upstream_addr $upstream_response_time $status $bytes_sent $proxy_add_x_forwarded_for $request_uri';
access_log /usr/local/nginx/logs/cache.log cache;
upstream origin {
server <origin-domain>;
}
server {
listen 80;
server_name localhost;
location ~ \.mp4$ {
mp4;
proxy_cache icdn_cache;
proxy_pass http://origin;
proxy_cache_key $uri;
}
}
I also tried:
location / {
location ~ \.mp4$ { mp4; }
proxy_cache icdn_cache;
proxy_pass http://origin;
proxy_cache_key $uri;
}
Is there a way to make cached mp4-files work with the seeking-function of mp4-module?
You must use proxy_store. proxy_cache will create a lot of files for every ?start=xxxx request.
To let an mp4 module seek in files you need the full movie. proxy_store will make a mirror on the cache server.
proxy_cache is part of proxy module. Currently you can't use nginx mp4 module with proxy, it only works for static files, that's it.