AWS S3 Not Sending Access-Control-Allow-Origin header when Origin header on request is present - ajax

I have an AWS S3 bucket with the following CORS policy:
<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
<AllowedOrigin>*</AllowedOrigin>
<AllowedMethod>GET</AllowedMethod>
<AllowedHeader>*</AllowedHeader>
<ExposeHeader>Access-Control-Allow-Origin</ExposeHeader>
</CORSRule>
</CORSConfiguration>
In my app, I load a page of images from the bucket. The images appear on the page as expected. When I click on one of the images to open it in the Adobe Creative SDK, the SDK fails to load the image because it is blocked by CORS. The SDK takes the url of the image and loads it via AJAX. The SDK has an option to enable CORS and appears to be sending the proper Origin header (See screenshot), however AWS is not including the Access-Control-Allow-Origin-Header, causing the image to be blocked by CORS.
I've scoured every question on here about this subject and it seems like no one has a straight answer. There are dozens of questions on this subject that are unanswered, however in many of those cases it appears that the AJAX request isn't sending the Origin header or that the bucket isn't properly configured. In this case, neither of those are true, which is why this is not a duplicate.
As you can see in the screenshot, the request includes the Origin header, but the response does not include the Access-Control-Allow-Origin header. What I would like to know is:
1. Why isn't S3 sending the proper header if my bucket is properly configured?
2. Is it possible that when the Adobe Creative SDK requests the image via AJAX, the browser sees that the image is already cached and tried to serve the cached image (which doesn't have the Origin header) instead?
3. If #2 is the case, how can I forcibly add the Origin header to the image request? In the app, the image is the background of a div (css background-image property).

I created a test bucket, enabled CORS, and used your CORS rules.
Note that I am not convinced that <ExposeHeader>Access-Control-Allow-Origin</ExposeHeader> is necessary, but it should not do any harm by being present, so I've retained it.
The behavior I observe in testing is not consistent with what you are showing in your browser's request/response headers.
Using curl, I set the Origin: header to http://example.com. (Yes, that's actually what I set it to... this was not modified in the output below).
$ curl -v http://xxxxxxxxxxxx.s3.amazonaws.com/index.txt -H 'Origin: http://example.com'
* Hostname was NOT found in DNS cache
* Trying 54.231.98.120...
* Connected to xxxxxxxxxxxx.s3.amazonaws.com (54.231.98.120) port 80 (#0)
> GET /index.txt HTTP/1.1
> User-Agent: curl/7.35.0
> Host: xxxxxxxxxxxx
> Accept: */*
> Origin: http://example.com
>
< HTTP/1.1 200 OK
< x-amz-id-2: pF39K26ii42SzxSU2Dt0KT2z7+xmfyiP4yekp9s4DCYJo0jlRwCTDg6QO6f0HMIL4H9b640zq7U=
< x-amz-request-id: 3B18A563CFF4E485
< Date: Sat, 28 May 2016 21:49:21 GMT
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: GET
< Access-Control-Expose-Headers: Access-Control-Allow-Origin
< Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
< Cache-Control: no-cache
< Last-Modified: Sat, 28 May 2016 21:40:57 GMT
< ETag: "cd21a8c7268dc6af728d180f9a3a81d7"
< Accept-Ranges: bytes
< Content-Type: text/plain
< Content-Length: 71
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
Why is this interesting?
S3, with CORS configured and a <CORSRule> matching your request, always returns the CORS response headers and a Vary: header (which means "If you vary one of the following headers in your request, I may vary something about my response.") The headers from the above output that I am referring to, specifically, are these.
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET
Access-Control-Expose-Headers: Access-Control-Allow-Origin
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
There is only one exception I can find to this, which is when there is not a matching CORS rule. To demonstrate that, I first changed my CORS configuration to <AllowedOrigin>http://example.com</AllowedOrigin> and then repeated the request. Note that the response is almost identical, except that S3 uses the exact origin in the response, and adds Access-Control-Allow-Credentials.
< Access-Control-Allow-Origin: http://example.com
< Access-Control-Allow-Methods: GET
< Access-Control-Expose-Headers: Access-Control-Allow-Origin
< Access-Control-Allow-Credentials: true
< Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
...however, if I -- still using the more restrictive CORS rule that specifies a specific origin -- then send a request for Origin: example.org, which is an origin that is not mentioned in my CORS configuration and has no wildcard <AllowedOrigin> to match, S3 responds as though CORS was not configured at all.
< HTTP/1.1 200 OK
< x-amz-id-2: WJ0QmIZ6jTQYefFi8GjlDQkZKFHDX8/5cmejeulhG1ov3/NdoSXbsTKetYpxvXPML8aUnPNZ/ac=
< x-amz-request-id: 15A03D8B1E08A830
< Date: Sat, 28 May 2016 21:58:34 GMT
< Cache-Control: no-cache
...etc.
This takes me back to my initial conclusion that the request you've shown your browser making does not match the timing of when you configured the bucket for CORS, and may have been cached before your bucket's CORS settings were in the correct state, or before they matched those you've posted in the question.
If an Origin: header was present on that request, as the browser shows that it was, then S3 should have added the CORS response headers, whether or not the request was actually a cross-origin request.
Your next steps would be to try this with a new object at a new path that there is no possibility of being cached, or try the common browser cache-busting tactic of adding ?some-random=thing-here to the object's URL when making the request. Failing that, you should consider proving or disproving correct behavior by your bucket using a tool like curl that shows exactly what's happening in the request/response.

Related

amp-list from remote location?

I would like to use amp-list on a website that is otherwise valid AMP to avoid using an amp-iframe embed with JS. I just finished reading the cors docs by Google AMP at https://www.ampproject.org/docs/fundamentals/amp-cors-requests and am still confused - is it possible to have the json source for the amp-list from a remote domain?
The thing I need is to have a source of URLs+titles generated and updated outside of the main jekyll website because the main website takes too long to build.
I am testing it with a valid JSON and headers as follows and am getting nothing in console and the list is not rendered, so I presume what I am trying to do is not possible?
/source.json
Content-type: application/json; charset=utf-8
X-Frame-Options: ALLOW
Access-Control-Allow-Headers: Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: https://fakename.netlify.com/
AMP-Access-Control-Allow-Source-Origin: https://fakename.netlify.com/
Access-Control-Allow-Methods: POST, GET, OPTIONS
Access-Control-Expose-Headers: AMP-Access-Control-Allow-Source-Origin

Cloudfront - why is this not being "browser cached"?

Here's the curl -I response to my Javascript file:
HTTP/1.1 200 OK
Content-Type: text/javascript
Content-Length: 72640
Connection: keep-alive
Date: Sat, 18 Feb 2017 16:12:06 GMT
Cache-Control: 86400
Last-Modified: Wed, 15 Feb 2017 15:09:28 GMT
ETag: "a6ee06ff5e49a4290bb2aabe5e0f9029"
Server: AmazonS3
Vary: Accept-Encoding
Age: 1173
X-Cache: Hit from cloudfront
Via: 1.1 3b17302562f1709d8b6c9f7be1.cloudfront.net (CloudFront)
I can see the Cache-Control tag there. Not sure what the Vary and the ETag are doing, but so be it. Does this somehow specify to a user's browser NOT to cache this file? Why are Pingdom or Goog PageSpeed not recognising this as a browser-cacheable file?
Your Cache-Control header is present, but the value is not actually valid. The correct format looks like this:
Cache-Control: max-age=86400
The number, by itself, is meaningless.
ETag: is the entity tag -- an opaque value that uniquely identifies the current content of a given URL. If the content changes, the ETag will also change. A browser with a cached copy may use this value for subsequent requests to ask that the server only return the content if it differs, by sending an If-None-Match: request header, including the last-seen ETag.
Vary: tells the browser that certain changes to the request may generate a different response. Unlike browsers, curl doesn't advertise its ability to support gzipped payload unless you specify the --compressed option. Adding that option when invoking curl triggers the addition of Accept-Encoding: gzip to the request, which may trigger the response to be compressed if you have that option enabled in CloudFront.

Issue loading webfonts when served from AWS Cloudfront

So, here's my conundrum! I have been trying to get Cloudfront to play nicely with my nginx server for the past three days...have read countless StackOverflow posts and blog articles...scoured the interwebs and I am still stuck with issues surrounding cross-domain access policies when it comes to Cloudfront serving fonts. I am going to post my complete setup in hopes that someone with more expertise may help me figure out what is going on. In the future, I hope this post will serve many others facing similar issues. Here goes...
Nginx config:
I have a nginx webserver with the following server block configuration. (...truncated for brevity)
server {
server_name example.com www.example.com;
root /var/www/example.com/html;
index index.html index.htm;
location / {
try_files $uri $uri/ =404;
}
location /assets {
autoindex on;
}
# Media
location ~* \.(jpe?g|gif|png|ico|cur|gz|svgz?|mp4|ogg|ogv|webm|htc|webp)$ {
expires 1M;
access_log off;
add_header Cache-Control public;
}
# Fonts
location ~* \.(eot|ttf|woff|woff2|svg)$ {
expires 365d;
access_log off;
add_header Cache-Control public;
add_header Access-Control-Allow-Origin example.com;
// Have also tried setting the "Access-Control-Allow-Origin" header to "*", but I'd prefer not to do this for security reasons.
}
}
FYI: All of my website files I would like to serve and offload to CloudFront are in the /assets directory of my virtual host. (ie. http://example.com/assets/..)
CloudFront:
I have created a new CloudFront distribution with the following settings:
Note: I am not using S3 to host my website files and assets.
General:
Alternate Domain Names (CNAMEs): static.example.com
Default Root Object: index.html
Origins:
Origin Domain Name: example.com
Origin Path: (left blank)
Origin SSL Protocols: TLSv1.2, TLSv1.1, TLSv1
Origin Protocol Policy: HTTP Only
HTTP Port: 80
HTTPS Port: 443
Origin Custom Headers: (none)
Behaviors:
Path Pattern: Default ()*
Viewer Protocol Policy: HTTP and HTTPS
Allowed HTTP Methods: GET, HEAD
Cached HTTP Methods: GET, HEAD (Cached by default)
Forward Headers: Whitelist
Access-Control-Allow-Origin
Object Caching: Use Origin Cache Headers
Forward Query Strings: No (Improves Caching)
Compress Objects Automatically: Yes
DNS settings:
A portion of my Zone file...
example.com. 1800 IN A 12.34.567.890 //faked IP here for privacy reasons
www.example.com. 1800 IN CNAME example.com.
static.example.com. 1800 IN CNAME kg72kgf83nhfy3.cloudfront.net. //faked CloudFront dist. domain name here for privacy reasons
What's happening? Why it no- worky?
So, CloudFront does its thing processing and deploying my distribution, and I'm assuming pulling the assets from my '/assets/..' web directory. All src, href, and CSS url() references point to http://static.example.com in my current HTML and CSS documents, including all font-face references. After the distribution is deployed, I hit my site http://example.com in a browser.
It appears that all static site assets are served correctly from CloudFront w/ appropriate caching headers as defined in my nginx config...EXCEPT...webfonts. I am getting missing fonts on the page and cross-domain access policy error messages in my browser console.
Headers—
An image, for reference (when pinging my server):
curl -I "http://example.com/assets/images/image1.png"
HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Sun, 15 May 2016 02:09:25 GMT
Content-Type: image/png
Content-Length: 194665
Last-Modified: Sun, 15 May 2016 01:52:35 GMT
Connection: keep-alive
ETag: "5737d663-2f869"
Expires: Tue, 14 Jun 2016 02:09:25 GMT
Cache-Control: max-age=2592000
Cache-Control: public
Accept-Ranges: bytes
A font (when pinging my server):
curl -I "http://example.com/assets/fonts/webfont.woff"
HTTP/1.1 200 OK
Server: nginx/1.6.3
Date: Sun, 15 May 2016 02:10:29 GMT
Content-Type: application/font-woff
Content-Length: 8752
Last-Modified: Sun, 15 May 2016 01:51:55 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "5737d63b-2230"
Expires: Mon, 15 May 2017 02:10:29 GMT
Cache-Control: max-age=31536000
Cache-Control: public
Access-Control-Allow-Origin: example.com
Accept-Ranges: bytes
Same image (when requesting from CloudFront):
curl -I "http://static.example.com/assets/images/image1.png"
HTTP/1.1 200 OK
Content-Type: image/png
Content-Length: 194665
Connection: keep-alive
Server: nginx/1.6.3
Date: Sun, 15 May 2016 02:39:16 GMT
Last-Modified: Sun, 15 May 2016 01:52:35 GMT
ETag: "5737d663-2f869"
Expires: Tue, 14 Jun 2016 02:39:16 GMT
Cache-Control: max-age=2592000
Cache-Control: public
Accept-Ranges: bytes
X-Cache: Miss from cloudfront
Via: 1.1 lots_of_random_characters_i_dont_know_if_should_share_here.cloudfront.net (CloudFront)
X-Amz-Cf-Id: lSM1plINYENbYycBn424LJ2wdtDhS3CpqAFiDSoxQDEctP_WM09bUQ==
Same font (when requesting from CloudFront):
curl -I "http://static.example.com/assets/fonts/webfont.woff"
HTTP/1.1 200 OK
Content-Type: application/font-woff
Content-Length: 8752
Connection: keep-alive
Server: nginx/1.6.3
Date: Sun, 15 May 2016 02:41:00 GMT
Last-Modified: Sun, 15 May 2016 01:51:55 GMT
ETag: "5737d63b-2230"
Expires: Mon, 15 May 2017 02:41:00 GMT
Cache-Control: max-age=31536000
Cache-Control: public
Access-Control-Allow-Origin: example.com
Accept-Ranges: bytes
Vary: Accept-Encoding
X-Cache: Miss from cloudfront
Via: 1.1 lots_of_random_characters_i_dont_know_if_should_share_here.cloudfront.net (CloudFront)
X-Amz-Cf-Id: vNfiyurS8pjosofnpLNSrnZuaGFg0V4xIs4ySCm05NKDMZ_PozhuOg==
Loading my website at http://example.com, everything seems to work (ie. images) EXCEPT the webfonts. Checking the browser console outputs the following message for every font:
Font from origin 'http://static.example.com' has been blocked from loading by Cross-Origin Resource Sharing policy: The 'Access-Control-Allow-Origin' header contains the invalid value 'example.com'. Origin 'http://example.com' is therefore not allowed access.
So, anyone have any thoughts?? I would greatly appreciate the help/input. I'm a young web developer just trying to learn.
Thank you! :)
–Kyle
Footnotes:
One of many blog posts I've followed yet haven't been able to get things working.
My plan is to append hashed query strings to the end of every file (ie. //static.example.com/assets/images/image.png?622c6911) to invalidate the CloudFront cache. This way I do not have to always re-upload new assets with changing names...I can simply control invalidation from the HTML and append a new query string to assets when I want CloudFront to request the latest version of that file from my webserver Origin.
I will eventually be purchasing a SSL certificate for my site, so I would like traffic to support both HTTP and HTTPS requests.
The explanation is here:
The 'Access-Control-Allow-Origin' header contains the invalid value 'example.com'. Origin 'http://example.com' is therefore not allowed access.
The origin is http://example.com... not example.com. Your response header has the wrong value. An origin is, by definition, scheme + hostname + port (with the implicit ports 80 and 443 omitted for http and https on standard ports).
When the request isn't subject to cross-origin rules, such as is the case with images, this misconfiguration on your web server is being ignored by the browser. For fonts, you're hitting this wall because the value is malformed.
add_header Access-Control-Allow-Origin example.com;
This is the root of your problem. You need to respond with the same origin sent by the browser in the Origin: header, assuming that origin is indeed valid and should be allowed. I'm not an nginx specialist, by according to this answer, you can validate the incoming origin with a regex, and set the response accordingly:
if ($http_origin ~* "^https?://.*example\.com$" ) {
add_header Access-Control-Allow-Origin $http_origin;
}
Not being familiar with the quirks of nginx regexes, mine is a little permissive, but you get the idea.
Before this will work, we need to fix the CloudFront cache behavior.
Forward Headers: Whitelist
Access-Control-Allow-Origin
That isn't a request header... that's a response header, so whitelisting it doesn't actually do anything.
You will, instead, need to whitelist the Origin header so that it will be forwarded to the web server by CloudFront, so that the server can respond with that same value in Access-Control-Allow-Origin:, as illustrated above.
Access-Control-Request-Headers and Access-Control-Request-Method should probably also be whitelisted, though for GET requests, I don't know that they will matter.
You also should probably modify the allowed methods to include OPTIONS in addition to GET and HEAD.
Bonus material:
My plan is to append hashed query strings to the end of every file (ie. //static.example.com/assets/images/image.png?622c6911) to invalidate the CloudFront cache.
In that case, you need Forward Query Strings set to Yes.
CloudFront caches objects based on what it actually sends to the server. Appending a query string will only invalidate the browser cache, not the CloudFront cache, if query strings are not forwarded to the origin. Yes, you need this enabled even if your server doesn't want or need the query strings, if the query strings are intended to be used for cache-busting CloudFront. That's why not forwarding them "improves caching." It will be "improved" by CloudFront ignoring them entirely when determining whether it already has a cached version of an object.
Also, note that the Via header doesn't contain sensitive information, neither does X-Amz-Cf-Id.

Does if-no-match need to be set programmatically in ajax request, if server sends Etag

My question is pretty simple. Although while searching over, I have not found a simple satisfying answer.
I am using Jquery ajax request to get the data from a server. Server
hosts a rest API that sets the Etag and Cach-control headers to the GET requests. The Server also sets CORS headers to allow the Etag.
The client of the Api is a browser web app. I am using Ajax request to call the Api. Here are the response headers from server after a simple GET request:
Status Code: 200 OK
Access-Control-Allow-Origin: *
Cache-Control: no-transform, max-age=86400
Connection: Keep-Alive
Content-Encoding: gzip
Content-Type: application/json
Date: Sun, 30 Aug 2015 13:23:41 GMT
Etag: "-783704964"
Keep-Alive: timeout=15, max=99
Server: Apache-Coyote/1.1
Transfer-Encoding: chunked
Vary: Accept-Encoding
access-control-allow-headers: X-Requested-With, Content-Type, Etag,Authorization
access-control-allow-methods: GET, POST, DELETE, PUT
All I want to know is:
Do I need to manually collect the Etag from response headers sent from the server and attach an if-no-match header to ajax request?OR the Browser sends it by-default in a conditional get request when it has an 'Etag'
I have done debugging over the network console in the browser and It
seems the browser is doing the conditional GET automatically and
sets the if-no-match header.
if it is right, Suppose, I created a new resource, and then I called the get request. It gives me the past cached data for the first time. But when I reload the page, It gives the updated one. So I am confused that, If the dataset on the server-side has changed and it sends a different Etag, Why doesn't the browser get an updated data set from the server unless I have to reload
Also in case of pagination. Suppose I have a URL /users?next=0. next is a query param where the value for the next changes for every new request. Since each response will get its own 'Etag'. Will the browser store the 'Etag' based on request or it just stores the lastest Etag of the previous get request, irrespective of the URL.
Well, I have somehow figured out the solution myself:
The browser sends the if-no-match header itself when it sees url had the e-tag header on a previous request. Browser saves the e-tag with respect to that URL, so it does not matter how many requests with different URLs happen.
Also, a trick to force the browser to fetch a conditional-get to check the e-tag:
Set the max-age header to the lowest (for me 60s works great)
once the cache expires, thebrowser will send a conditional-get to check if the expired cached resource is valid. If the if-no-match header matches with e-tag. The server sends the response back with 304: Not-Modified header. This means the expired cached resource is valid and can be used.

Cannot get CORS working when trying to list bucket contents on Google Cloud Storage (404 not found)

I followed all the instructions and 'troubleshooting' guide for CORS but I have a feeling it's just not working.. i've spent the last 2 days just trying to list a bucket contents and not matter how i configure my CORS for my bucket it fails same way.
What I'm using:
An angularJS client app talking directly to Google Cloud Storage using JSON API from my local machine (i.e. localhost)
Followed all the advice here:
https://developers.google.com/storage/docs/cross-origin
My Bucket:
gs://okrp-dev
THE ERROR (chrome latest browser)
XMLHttpRequest cannot load https://storage.googleapis.com/storage/v1beta2/b/okrp-dev/o. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'http://localhost' is therefore not allowed access.
My CORS file now (though i've tried every other permutation):
bash-3.2$ gsutil cors get gs://okrp-dev
<?xml version="1.0" ?>
<CorsConfig>
<Cors>
<Origins>
<Origin>*</Origin>
<Origin>
http://localhost</Origin>
</Origins>
<Methods>
<Method>GET</Method>
<Method>POST</Method>
<Method>HEAD</Method>
</Methods>
<ResponseHeaders>
<ResponseHeader>*</ResponseHeader>
</ResponseHeaders>
<MaxAgeSec>86400</MaxAgeSec>
</Cors>
</CorsConfig>
Trace from Chrome dev tools Network tab:
Remote Address:74.125.193.132:443
Request URL:https://storage.googleapis.com/storage/v1beta2/b/okrp-dev/o
Request Method:OPTIONS
Status Code:200 OK
Request Headers
:host:
storage.googleapis.com
:method:OPTIONS
:path: /storage/v1beta2/b/okrp-dev/o
:scheme:https
:version:
HTTP/1.1
accept:*/*
accept-encoding:
gzip,deflate,sdch
accept-language:en-US,en;q=0.8,tr;q=0.6
access-control-request-headers:access-control-allow-origin, accept, authentication
access-control-request-method:GET
cache-control:
no-cache
origin:
http://localhost
pragma:
no-cache
referer:http://localhost/okrp/app/
Response Headersview source
alternate-protocol:443:quic
cache-control:
private, max-age=0
content-length:0
content-type:
text/html; charset=UTF-8
server:HTTP Upload Server Built on Mar 5 2014 15:51:04 (1394063464)
status:200 OK
version:
HTTP/1.1
NOTE I have the authentication working fine. I'm no longer getting a 401. Only 404.
NOTE I also of course totally opened up all permission on the bucket and object and that does not help at all.
How else can I troubleshoot this??

Resources