I am using Packetbeat to monitor the requests/responses into/out of Elasticsearch client nodes using the http protocol watcher on port 9200. I am sending the output of Packetbeat through Logstash, and then from there out to a different instance of Elasticsearch. We have compression support enabled in the Elasticsearch that is being monitored, so I occasionally see requests with "Accept-Encoding: gzip, deflate" headers returning responses that are gzipped. Unfortunately, I have not been able to decode any of these gzip responses using any tools I have at my disposal (including the web-based converters, the gzip command line tool, and using Zlib::GzipReader in a Logstash ruby filter script). They all report that it is not a gzip format.
Does anyone know why I can't seem to decode the gzip content?
I have provided a sample of the filter I'm using in Logstash to try to do this on the fly as the event passes through Logstash (and it always reports that http.response.body is not in gzip format).
filter {
if [type] == "http" {
if [http][response][headers][content-encoding] == "gzip" {
ruby {
init => "
require 'zlib'
require 'stringio'
"
code => "
body = event.get('[http][response][body]').to_s
sio = StringIO.new(body)
gz = Zlib::GzipReader.new(sio)
result = gz.read.to_s
event.set('[http][response][body]', result)
"
}
}
}
}
I'm also providing a sample of the logged event here which includes the gzip content in case you would like to try to decompress it yourself:
{
"_index": "packetbeat-6.2.3-2018.05.19",
"_type": "doc",
"_id": "oH0bemMB2mAXfg5euIiP",
"_score": 1,
"_source": {
"server": "",
"client_server": "",
"bytes_in": 160,
"bytes_out": 361,
"#timestamp": "2018-05-19T20:33:46.470Z",
"client_port": 55863,
"path": "/",
"type": "http",
"client_proc": "",
"query": "GET /",
"port": 9200,
"host": "gke-main-production-elastic-clients-5728bab3-t1z8",
"#version": "1",
"responsetime": 0,
"fields": {
"nodePool": "production-elastic-clients"
},
"response": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-encoding: gzip\r\ncontent-length: 250\r\n\r\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"proc": "",
"request": "GET / HTTP/1.1\r\nUser-Agent: vscode-restclient\r\nhost: es-http-dev.elastic-prod.svc.cluster.local:9200\r\naccept-encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
"beat": {
"name": "gke-main-production-elastic-clients-5728bab3-t1z8",
"version": "6.2.3",
"hostname": "gke-main-production-elastic-clients-5728bab3-t1z8"
},
"status": "OK",
"method": "GET",
"client_ip": "10.24.20.6",
"http": {
"response": {
"phrase": "OK",
"headers": {
"content-encoding": "gzip",
"content-length": 250,
"content-type": "application/json; charset=UTF-8"
},
"body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"code": 200
},
"request": {
"params": "",
"headers": {
"connection": "keep-alive",
"user-agent": "vscode-restclient",
"content-length": 0,
"host": "es-http-dev.elastic-prod.svc.cluster.local:9200",
"accept-encoding": "gzip, deflate"
}
}
},
"tags": [
"beats",
"beats_input_raw_event"
],
"ip": "10.24.41.5"
},
"fields": {
"#timestamp": [
"2018-05-19T20:33:46.470Z"
]
}
}
And this is the response for that message that I receive at the client after it has been decompressed successfully by the client:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 250
{
"name": "es-client-7688c8d9b9-qp9l7",
"cluster_name": "esprod",
"cluster_uuid": "8iRwLMMSR72F76ZEONYcUg",
"version": {
"number": "5.6.3",
"build_hash": "1a2f265",
"build_date": "2017-10-06T20:33:39.012Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
I had a different situation and was able to resolve my issue. Posting it here, see if it helps your case.
I was using postman tool to test my REST API services locally. My Packetbeat used following config.
type: http
ports: [80, 8080, 8000, 5000, 8002]
send_all_headers: true
include_body_for: ["application/json", "x-www-form-urlencoded"]
send_request: true
send_response: true
I was getting following output in body.
I was able to get http.response.body in clear text when i added following to my postman request.
Accept-Encoding: application/json
Quick question re: a 401 error I'm running into when executing a report.getfile in the Doubleclick Search API Explorer (full error message below) .
I am currently logged into my google account that is used for Doubleclick Search. I've tried logging in on a different browser, as well.
I've gone through the 'report.request' and 'report.get' executions without issue. I only get this error when I get to this point. "isReportReady" is also true before I execute 'report.getfile'. Full list of requests/responses below this initial message.
My understanding is that with API Explorer, I should just be able to turn on the OAUTH2.0 and execute, but correct me if I'm wrong!
"domain": "global",
"reason": "required",
"message": "Login Required",
"locationType": "header",
"location": "Authorization"
"code": 401,
"message": "Login Required"
If any additional details are needed definitely let me know. The only questions I've seen come up in here seem to be more-so based around manual coding, or other API Explorer services.
Thanks in advance for any insight!
reports.Request (Response)
POST https://www.googleapis.com/doubleclicksearch/v2/reports?fields=isReportReady%2Crequest%2Fcolumns%2FcolumnName&key={YOUR_API_KEY}
{
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"reportType": "conversion",
"statisticsCurrency": "agency",
"reportScope": {
"agencyId": "xxxxxxxxxxxxxxxxxxxx",
"advertiserId": "xxxxxxxxxxxxxxxxxxx"
},
"timeRange": {
"startDate": "2014-01-1",
"endDate": "2016-11-3"
},
"columns": [
{
"columnName": "conversionVisitExternalClickId"
},
{
"columnName": "conversionTimestamp"
},
{
"columnName": "keywordid"
},
{
"columnName": "adGroup"
},
{
"columnName": "campaign"
},
{
"columnName": "account"
},
{
"columnName": "floodlightActivity"
},
{
"columnName": "conversionID"
}
]
}
reports.Get (Response)
"kind": "doubleclicksearch#report",
"id": "xxxxxxxxxxxxx",
"isReportReady": true,
"request": {
"reportScope": {
"agencyId": "xxxxxxxxxxxxxxx",
"advertiserId": "xxxxxxxxxxxxx"
},
"reportType": "conversion",
"columns": [
{
"columnName": "conversionVisitExternalClickId"
},
{
"columnName": "conversionTimestamp"
},
{
"columnName": "keywordId"
},
{
"columnName": "adGroup"
},
{
"columnName": "campaign"
},
{
"columnName": "account"
},
{
"columnName": "floodlightActivity"
},
{
"columnName": "conversionId"
}
],
"timeRange": {
"startDate": "2014-01-1",
"endDate": "2016-11-3"
},
"includeRemovedEntities": false,
"downloadFormat": "csv",
"statisticsCurrency": "agency",
"maxRowsPerFile": 6000000
},
"statisticsCurrencyCode": "USD",
"statisticsTimeZone": "America/New_York",
"rowCount": 5225201,
"files": [
{
"url": "https://www.googleapis.com/doubleclicksearch/v2/reports/AAAncMCcKrSJ9MS_/files/0",
"byteCount": "1271002762"
}
]
}
reports.getFile (Response)
GET https://www.googleapis.com/doubleclicksearch/v2/reports/AAAncMCcKrSJ9MS_/files/0?key={YOUR_API_KEY}
200
- Hide headers -
cache-control: private, max-age=0, must-revalidate, no-transform
content-length: 1694670352
Content-Type: text/csv
date: Tue, 21 Mar 2017 14:24:03 GMT
etag: W/"u6ybyji1w_vhYZMTUlOUhEWemRI/nWLUgbdAKD9EU6srfscDcZyGxz8"
expires: Tue, 21 Mar 2017 14:24:03 GMT
server: UploadServer
vary: Origin, X-Origin
x-guploader-uploadid: AEnB2UoZzz4AafhGUKg-1edKZWLCmiML_a6HnP7lr11aD7tXZAnEIgyEno9ZhHOCbJ02XTlAkgtYcD4AOGbhmar6dpSx-uzh-n_6SCQH-iShQma-k6GLsMk
Why is cloudfront responding with a 416 status in the following scenario, even though I believe the requested range should be satisfiable?
Original S3 object
The scenario:
configure cloudfront distribution with an S3 origin, Object Caching set to "Customize", min/max/default TTL all set to 0
upload a video/mp4 file to S3, with no cache related headers set
playback video in chrome, via cloudfront
At this point, everything seems fine. I see 3 network requests to the resource - why chrome is requesting overlapping ranges in 2 and 3, I don't understand, but nevertheless, the video plays fine at this point.
Overwritten S3 object
Now, soon after:
upload larger (~ 2x) file to same S3 location
playback video in chrome, via cloudfront
This time, an error occurs.
It seems as though resp 1 here is ok (i.e. content length looks to be correct) - although not sure why it's a Miss from cloudfront - I would've expected it to be a RefreshHit given the TTL settings
Why is "req 2" of second page load failing with a 416, even though the requested range is within the content length from resp 1?
Notes:
going direct to S3, there is no problem
replacing the bigger file back to the smaller file, the video plays fine again
uploading the file to S3 with "cache-control: no-cache", there is no problem but I always get a "Miss from cloudfront". My understanding is that cloudfront should always be checking with S3 to see if content has changed with this configuration
waiting for some time (say 30s) - the problem resolves itself
previous attempts at replicating this scenario saw an addition 304 response from S3 to CloudFront after the overwrite, but I haven't been able to replicate that exact behaviour (there is just a single 216 now)
Logs
Original S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "535351" },
{ "name": "x-amz-cf-id", "value": "-6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=524288-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 524288-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "11063" },
{ "name": "x-amz-cf-id", "value": "_z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=32768-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 32768-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "502583" },
{ "name": "x-amz-cf-id", "value": "8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ==" }
]
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:23:53 MEL50 34771 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - Error -6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw== dvayusv1lektq.cloudfront.net https 46 1.613 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:23:56 MEL50 11408 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit _z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A== dvayusv1lektq.cloudfront.net https 47 2.194 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
2016-12-11 23:23:56 MEL50 503468 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit 8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ== dvayusv1lektq.cloudfront.net https 47 0.259 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:44 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> FB18FDABAA5DF6CA REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 535351 6448 118 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:53 +0000] 54.239.202.78 - 934DB75AC20953C3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 535351 535351 101 98 "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:54 +0000] 54.239.202.78 - 56EF7496F985D4B3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 11 - "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:56 +0000] 54.239.202.78 - 7F0087B4769D0FD3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 4 - "-" "Amazon CloudFront" -
Overwritten S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:24:07 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-956851/956852" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "956852" },
{ "name": "x-amz-cf-id", "value": "CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 416,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "range", "value": "bytes=917504-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "server", "value": "CloudFront" },
{ "name": "x-cache", "value": "Error from cloudfront" },
{ "name": "content-type", "value": "text/html" },
{ "name": "status", "value": "416" },
{ "name": "content-length", "value": "49" },
{ "name": "x-amz-cf-id", "value": "WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA==" },
{ "name": "expires", "value": "Sun, 11 Dec 2016 23:24:19 GMT" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 0,
"request_headers": [
{ "name": "User-Agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "Range", "value": "bytes=32768-" },
{ "name": "Accept-Encoding", "value": "identity;q=1, *;q=0" }
],
"response_headers": []
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:24:19 MEL50 52198 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A== dvayusv1lektq.cloudfront.net https 46 1.551 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:24:19 MEL50 302 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 416 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA== dvayusv1lektq.cloudfront.net https 47 0.001 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Error HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:06 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> E58826B8A66DBA1B REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 956852 7367 53 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:18 +0000] 54.239.202.45 - BA1C06FEA7DAC83F REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 956852 956852 40 35 "-" "Amazon CloudFront" -
What you can't see is which object -- the old one or the new one -- S3 is serving to CloudFront on the first requests after the overwrite.
When you overwrite an existing object in S3, the overwrite is always an atomic operation, in the sense that every request will be satisfied with either the complete old object or the complete new object... but the timing is not guaranteed, because S3 has an eventual consistency model for overwrites of existing objects.
Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all regions.
Updates to a single key are atomic. For example, if you PUT to an existing key, a subsequent read might return the old data or the updated data, but it will never write corrupted or partial data.
http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
Even if the bucket does not have versioning enabled, there's a window of time -- usually very short -- when both objects exist in S3. It can't be any other way, given that the above statement is true.
Similarly, it's possible to request a nonexistent object from S3, then upload the object, then request it again and continue to get a 404 (or 403, depending on bucket configuration) response for a short time before downloads succeed.
Conversely, if there has never been an attempt to download a nonexistent object, you will always find it immediately available for download after you upload it. This is a tradeoff that is essentially a design necesssity at large scale.
The bucket and distribution logs should help reveal what's happening behind the scenes, particularly if the CloudFront download from S3 on any of the download attempts after the upload shows a byte count consistent with the old object.
Upon further reflection... there is a second possible explanation what is happening, here, and it is difficult to decide whether or not this might be considered a bug in CloudFront, if indeed this is the actual issue.
After the object is replaced in S3, let's assume for a moment that consistency is not an issue -- let's assume thst all subsequent requests after the upload do in fact cause S3 to provide the current version of the object to CloudFront.
The problem here may be that the first download after the object was replaced -- which should have caused CloudFront to evict its old cached version of the object -- is being canceled by the browser. (I've seen Chrome do this, though I don't know exactly why it does it.)
When a download is canceled, the response from the origin is not cached.
Canceled Requests
If an object is not in the edge cache, and if a viewer terminates a session (for example, closes a browser) after CloudFront gets the object from your origin but before it can deliver the requested object, CloudFront does not cache the object in the edge location.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorS3Origin.html#response-s3-canceled-requests
Granted, this says "if the object is not in the edge cache," and you could argue that it actually is... but it may be a question of semantics: the object CloudFront is requesting from S3 is arguably not in the cache -- what's in the cache is a different object (at the same URI). If this logic holds, then the assertion that "CloudFront does not cache the object" presumably still holds, as well.
So... request 2 asks for a range outside of the range of the old object, which is the only thing CloudFront knows about. CloudFront dutifully reports the fact that -- for all it knows -- the request is indeed out of bounds, requested range not satisfiable.
Arguably, CloudFront should verify that the object is fresh, first... but also arguably, the browser should not make a subsequent range request for an object it has never successfully downloaded.
Maybe we can shed some light on this with correlated log files.