Elasticsearch response time - strange values - elasticsearch

I'm querying simple Elasticsearch index with house numbers data.
".house-numbers": {
"mappings": {
"house-number": {
"properties": {
"id": {
"type": "keyword"
},
"value": {
"type": "text",
"index_options": "docs"
}
}
}
}
}
Then I'm querying data like POST http request
Request url
http://localhost:9200/.house-numbers/housenumber/_search
Headers:
Content-Type: text/plain
Content-Length: 55
Accept: */*
Accept-Encoding: gzip, deflate, br
Request body:
{
"size": 30,
"query": {
"match": {
"value": {
"query": "2 3"
}
}
}
}
Request returns data in 10ms - 30ms and everything works fine. Elasticsearch reponse parameter took is small in all cases 3-5ms.
When I change size in request body to "size": 35 response time has suddenly 500ms. Took parameter from Elasticsearch is the same. There are no special characters and size of response is very similar.
I tried many clients NEST, Postman, Fiddler to do these requests, every client has the same behaviour.
Setting of my elasticsearch contains only
http.compression : true
http.compression_level : 9
Setting of my jvm
"jvm": {
"timestamp": 1478108615141,
"uptime_in_millis": 17150141,
"mem": {
"heap_used_in_bytes": 1384307624,
"heap_used_percent": 66,
"heap_committed_in_bytes": 2077753344,
"heap_max_in_bytes": 2077753344,
"non_heap_used_in_bytes": 96403904,
"non_heap_committed_in_bytes": 101502976,
"pools": {
"young": {
"used_in_bytes": 324358632,
"max_in_bytes": 558432256,
"peak_used_in_bytes": 558432256,
"peak_max_in_bytes": 558432256
},
"survivor": {
"used_in_bytes": 69730304,
"max_in_bytes": 69730304,
"peak_used_in_bytes": 69730304,
"peak_max_in_bytes": 69730304
},
"old": {
"used_in_bytes": 990220848,
"max_in_bytes": 1449590784,
"peak_used_in_bytes": 1190046816,
"peak_max_in_bytes": 1449590784
}...
I tried different versions of elasticsearch
I tried different settings - turn off http.compression, change compression_level
I tried another hosts for elasticsearch
I have no idea what can cause this problem and I can't continue with my work.
Any idea where to look or how to proceed?

Of course problem was not in elasticsearch but in the http communication, especially when http compression was turned on
Hints to remove delays
close fiddler
disable firewall and anti-virus software
close all programs that possibly can intercept http communication

Related

Decoding gzip response body from Packetbeat

I am using Packetbeat to monitor the requests/responses into/out of Elasticsearch client nodes using the http protocol watcher on port 9200. I am sending the output of Packetbeat through Logstash, and then from there out to a different instance of Elasticsearch. We have compression support enabled in the Elasticsearch that is being monitored, so I occasionally see requests with "Accept-Encoding: gzip, deflate" headers returning responses that are gzipped. Unfortunately, I have not been able to decode any of these gzip responses using any tools I have at my disposal (including the web-based converters, the gzip command line tool, and using Zlib::GzipReader in a Logstash ruby filter script). They all report that it is not a gzip format.
Does anyone know why I can't seem to decode the gzip content?
I have provided a sample of the filter I'm using in Logstash to try to do this on the fly as the event passes through Logstash (and it always reports that http.response.body is not in gzip format).
filter {
if [type] == "http" {
if [http][response][headers][content-encoding] == "gzip" {
ruby {
init => "
require 'zlib'
require 'stringio'
"
code => "
body = event.get('[http][response][body]').to_s
sio = StringIO.new(body)
gz = Zlib::GzipReader.new(sio)
result = gz.read.to_s
event.set('[http][response][body]', result)
"
}
}
}
}
I'm also providing a sample of the logged event here which includes the gzip content in case you would like to try to decompress it yourself:
{
"_index": "packetbeat-6.2.3-2018.05.19",
"_type": "doc",
"_id": "oH0bemMB2mAXfg5euIiP",
"_score": 1,
"_source": {
"server": "",
"client_server": "",
"bytes_in": 160,
"bytes_out": 361,
"#timestamp": "2018-05-19T20:33:46.470Z",
"client_port": 55863,
"path": "/",
"type": "http",
"client_proc": "",
"query": "GET /",
"port": 9200,
"host": "gke-main-production-elastic-clients-5728bab3-t1z8",
"#version": "1",
"responsetime": 0,
"fields": {
"nodePool": "production-elastic-clients"
},
"response": "HTTP/1.1 200 OK\r\ncontent-type: application/json; charset=UTF-8\r\ncontent-encoding: gzip\r\ncontent-length: 250\r\n\r\n\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"proc": "",
"request": "GET / HTTP/1.1\r\nUser-Agent: vscode-restclient\r\nhost: es-http-dev.elastic-prod.svc.cluster.local:9200\r\naccept-encoding: gzip, deflate\r\nConnection: keep-alive\r\n\r\n",
"beat": {
"name": "gke-main-production-elastic-clients-5728bab3-t1z8",
"version": "6.2.3",
"hostname": "gke-main-production-elastic-clients-5728bab3-t1z8"
},
"status": "OK",
"method": "GET",
"client_ip": "10.24.20.6",
"http": {
"response": {
"phrase": "OK",
"headers": {
"content-encoding": "gzip",
"content-length": 250,
"content-type": "application/json; charset=UTF-8"
},
"body": "\u001f�\b\u0000\u0000\u0000\u0000\u0000\u0000\u0000T��n�0\u0014Fw���\u001c\u0010\u0018�����&��vH\u0016d�K������\u0010��\u000b�C\u0018����{��\u0010]\u0001�\u001aap1W\u0012�\u0018\u0017�,y)���oC�\n��A��\u001b�6/��\u001a�\u000e��\"l+�����\u001d\u000f\u0005y/���k�?�\u0005�\u0005���3���Y�_[���Mh�\u0007nzo�T����C�1�\u0011�]����\u0007H�\u0015q��)�&i��u^%iF�k�i6�ތs�c���)�9hh^�0�T2<�<���.J����x���}�:c�\u0011��=���\u001f\u0000\u0000\u0000��\u0003\u0000��.�S\u0001\u0000\u0000",
"code": 200
},
"request": {
"params": "",
"headers": {
"connection": "keep-alive",
"user-agent": "vscode-restclient",
"content-length": 0,
"host": "es-http-dev.elastic-prod.svc.cluster.local:9200",
"accept-encoding": "gzip, deflate"
}
}
},
"tags": [
"beats",
"beats_input_raw_event"
],
"ip": "10.24.41.5"
},
"fields": {
"#timestamp": [
"2018-05-19T20:33:46.470Z"
]
}
}
And this is the response for that message that I receive at the client after it has been decompressed successfully by the client:
HTTP/1.1 200 OK
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 250
{
"name": "es-client-7688c8d9b9-qp9l7",
"cluster_name": "esprod",
"cluster_uuid": "8iRwLMMSR72F76ZEONYcUg",
"version": {
"number": "5.6.3",
"build_hash": "1a2f265",
"build_date": "2017-10-06T20:33:39.012Z",
"build_snapshot": false,
"lucene_version": "6.6.1"
},
"tagline": "You Know, for Search"
}
I had a different situation and was able to resolve my issue. Posting it here, see if it helps your case.
I was using postman tool to test my REST API services locally. My Packetbeat used following config.
type: http
ports: [80, 8080, 8000, 5000, 8002]
send_all_headers: true
include_body_for: ["application/json", "x-www-form-urlencoded"]
send_request: true
send_response: true
I was getting following output in body.
I was able to get http.response.body in clear text when i added following to my postman request.
Accept-Encoding: application/json

How to partial match response in the artillery tool?

I am using artillery tool for websocket testing. The problem I am facing is with response. After emitting data to the channel that I am testing, we always get different data. As artillery's Response block tries to match completely with the actual response from the channel (i.e It doesn't match partially) , I am always getting a error 'data is not valid: 1'. How to match partially the response?
Below response code is working.
{
"response": {
"channel": "channel1",
"match": [
{
"json": "$.Id",
"value": "test_101"
},
{
"json": "$.param1",
"value": "1"
},
{
"json": "$.param2",
"value": "XXX"
},
{
"json": "$.param3",
"value": "100"
}
]
}
}

Query on number field matching string fields

I have an ELK stack setup. When I am performing a query on number fields then it is also matching against string fields. For example, I am sending Load Balancer logs to ELK and if I perform backend_processing_time:>5 on that then it is matching against backend_processing_time with value 0.001 too.
On kibana interface, it is showing that the query is matching string in the request message. I am not able to understand how a query against a number field is matching against a string.
In the dev tools section on kibana i tried to run the same query
GET _search
{
"query": {
"range" : {
"backend_processing_time" : {
"gte" : 50000000000
}
}
}
}
Even with so much backend_processing_time i am getting results. I am not able to understand why this is happening.
I searched on other fields also which are of number type and found that all the queries done on number field are getting matched with string type fields.
I am providing a sample search result which i get for backend_processing_time:>500000000 query. It can be seen in this result that backend_processing_time field is so small but still getting a hit.
{
"_index": "logstash-2017.05.10",
"_type": "prod-quizelb-logs",
"_id": "AVvzYRgL49GPTZAKoDer",
"_score": null,
"_source": {
"backendport": 80,
"received_bytes": 0,
"request": "http://en.meaww.com:80/locales/en.json",
"backend_response": 200,
"verb": "GET",
"message": "2017-05-10T17:19:52.881044Z Prod-ELB 172.68.144.71:34803 10.1.91.253:80 0.000075 0.000606 0.000019 200 200 0 1881 \"GET http://en.meaww.com:80/locales/en.json HTTP/1.1\" \"Mozilla/5.0 (Linux; Android 6.0.1; SM-C900F Build/MMB29M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/58.0.3029.83 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/122.0.0.17.71;]\" - -\n",
"type": "prod-quizelb-logs",
"clientport": 34803,
"request_processing_time": 0.000075,
"urihost": "en.meaww.com:80",
"response_processing_time": 0.000019,
"path": "/locales/en.json",
"#timestamp": "2017-05-10T17:21:18.280Z",
"port": "80",
"response": 200,
"bytes": 1881,
"clientip": "172.68.144.71",
"proto": "http",
"#version": "1",
"elb": "Prod-ELB",
"httpversion": "1.1",
"backendip": "10.1.91.253",
"backend_processing_time": 0.000606,
"timestamp": "2017-05-10T17:19:52.881044Z"
},
"fields": {
"#timestamp": [
1494436878280
],
"timestamp": [
1494436792881
]
},
"highlight": {
"backend_processing_time.keyword": [
"#kibana-highlighted-field#6.06E-4#/kibana-highlighted-field#"
],
"request": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#://#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field#"
],
"elb.keyword": [
"#kibana-highlighted-field#Prod-ELB#/kibana-highlighted-field#"
],
"urihost.keyword": [
"#kibana-highlighted-field#en.meaww.com:80#/kibana-highlighted-field#"
],
"verb": [
"#kibana-highlighted-field#GET#/kibana-highlighted-field#"
],
"request.keyword": [
"#kibana-highlighted-field#http://en.meaww.com:80/locales/en.json#/kibana-highlighted-field#"
],
"type": [
"#kibana-highlighted-field#prod#/kibana-highlighted-field#-#kibana-highlighted-field#quizelb#/kibana-highlighted-field#-#kibana-highlighted-field#logs#/kibana-highlighted-field#"
],
"message": [
"2017-05-10T17:19:#kibana-highlighted-field#52.881044Z#/kibana-highlighted-field# #kibana-highlighted-field#Prod#/kibana-highlighted-field#-#kibana-highlighted-field#ELB#/kibana-highlighted-field# 172.68.144.71:34803 10.1.91.253:#kibana-highlighted-field#80#/kibana-highlighted-field# 0.000075 0.000606 0.000019 200 200 0 1881 \"#kibana-highlighted-field#GET#/kibana-highlighted-field# #kibana-highlighted-field#http#/kibana-highlighted-field#://#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field# #kibana-highlighted-field#HTTP#/kibana-highlighted-field#/1.1\" \"#kibana-highlighted-field#Mozilla#/kibana-highlighted-field#/5.0 (#kibana-highlighted-field#Linux#/kibana-highlighted-field#; #kibana-highlighted-field#Android#/kibana-highlighted-field# #kibana-highlighted-field#6.0.1#/kibana-highlighted-field#; #kibana-highlighted-field#SM#/kibana-highlighted-field#-#kibana-highlighted-field#C900F#/kibana-highlighted-field# #kibana-highlighted-field#Build#/kibana-highlighted-field#/#kibana-highlighted-field#MMB29M#/kibana-highlighted-field#; #kibana-highlighted-field#wv#/kibana-highlighted-field#) #kibana-highlighted-field#AppleWebKit#/kibana-highlighted-field#/#kibana-highlighted-field#537.36#/kibana-highlighted-field# (#kibana-highlighted-field#KHTML#/kibana-highlighted-field#, #kibana-highlighted-field#like#/kibana-highlighted-field# #kibana-highlighted-field#Gecko#/kibana-highlighted-field#) #kibana-highlighted-field#Version#/kibana-highlighted-field#/4.0 #kibana-highlighted-field#Chrome#/kibana-highlighted-field#/#kibana-highlighted-field#58.0.3029.83#/kibana-highlighted-field# #kibana-highlighted-field#Mobile#/kibana-highlighted-field# #kibana-highlighted-field#Safari#/kibana-highlighted-field#/#kibana-highlighted-field#537.36#/kibana-highlighted-field# [#kibana-highlighted-field#FB_IAB#/kibana-highlighted-field#/#kibana-highlighted-field#FB4A#/kibana-highlighted-field#;#kibana-highlighted-field#FBAV#/kibana-highlighted-field#/122.0.0.17.71;]\" - -\n"
],
"urihost": [
"#kibana-highlighted-field#en.meaww.com#/kibana-highlighted-field#:#kibana-highlighted-field#80#/kibana-highlighted-field#"
],
"path": [
"/#kibana-highlighted-field#locales#/kibana-highlighted-field#/#kibana-highlighted-field#en.json#/kibana-highlighted-field#"
],
"verb.keyword": [
"#kibana-highlighted-field#GET#/kibana-highlighted-field#"
],
"proto.keyword": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#"
],
"port": [
"#kibana-highlighted-field#80#/kibana-highlighted-field#"
],
"type.keyword": [
"#kibana-highlighted-field#prod-quizelb-logs#/kibana-highlighted-field#"
],
"proto": [
"#kibana-highlighted-field#http#/kibana-highlighted-field#"
],
"elb": [
"#kibana-highlighted-field#Prod#/kibana-highlighted-field#-#kibana-highlighted-field#ELB#/kibana-highlighted-field#"
],
"backend_processing_time": [
"#kibana-highlighted-field#6.06E#/kibana-highlighted-field#-4"
],
"port.keyword": [
"#kibana-highlighted-field#80#/kibana-highlighted-field#"
]
},
"sort": [
1494436878280
]
}
EDIT
I got the mapping by running GET /logstash-2017.05.11/_mapping/prod-quizelb-logs query in kibana console.
The mapping which I am getting for backend_processing_time is showing this
"backend_processing_time": {
"type": "text",
"norms": false,
"fields": {
"keyword": {
"type": "keyword"
}
}
}
So it seems that this field is of text type thus causing this error to happen.
Now I have another confusion i.e. kibana is showing this as number but elasticsearch is showing this of type text. Also, this is getting mapped dynamically as i never created the mapping on my own. I think that they are getting created by logstash at the time grok filter is applied.
You need to take control of the mapping of those index(indices) so that your field will actually be a number. Otherwise, you will not be sure what kind of field type you'll have there. So, basically you need something like this, either in an index template, or a static mapping all the way:
"backend_processing_time": {
"type": "integer"
}
Remove space in your query_string. i.e Your query_string should look like this:
backend_processing_time:>0.5
Read more about query_string syntax here

Doubleclick Search API Explorer: 'Login Required' for reports.GetFile

Quick question re: a 401 error I'm running into when executing a report.getfile in the Doubleclick Search API Explorer (full error message below) .
I am currently logged into my google account that is used for Doubleclick Search. I've tried logging in on a different browser, as well.
I've gone through the 'report.request' and 'report.get' executions without issue. I only get this error when I get to this point. "isReportReady" is also true before I execute 'report.getfile'. Full list of requests/responses below this initial message.
My understanding is that with API Explorer, I should just be able to turn on the OAUTH2.0 and execute, but correct me if I'm wrong!
"domain": "global",
"reason": "required",
"message": "Login Required",
"locationType": "header",
"location": "Authorization"
"code": 401,
"message": "Login Required"
If any additional details are needed definitely let me know. The only questions I've seen come up in here seem to be more-so based around manual coding, or other API Explorer services.
Thanks in advance for any insight!
reports.Request (Response)
POST https://www.googleapis.com/doubleclicksearch/v2/reports?fields=isReportReady%2Crequest%2Fcolumns%2FcolumnName&key={YOUR_API_KEY}
{
"downloadFormat": "csv",
"maxRowsPerFile": 6000000,
"reportType": "conversion",
"statisticsCurrency": "agency",
"reportScope": {
"agencyId": "xxxxxxxxxxxxxxxxxxxx",
"advertiserId": "xxxxxxxxxxxxxxxxxxx"
},
"timeRange": {
"startDate": "2014-01-1",
"endDate": "2016-11-3"
},
"columns": [
{
"columnName": "conversionVisitExternalClickId"
},
{
"columnName": "conversionTimestamp"
},
{
"columnName": "keywordid"
},
{
"columnName": "adGroup"
},
{
"columnName": "campaign"
},
{
"columnName": "account"
},
{
"columnName": "floodlightActivity"
},
{
"columnName": "conversionID"
}
]
}
reports.Get (Response)
"kind": "doubleclicksearch#report",
"id": "xxxxxxxxxxxxx",
"isReportReady": true,
"request": {
"reportScope": {
"agencyId": "xxxxxxxxxxxxxxx",
"advertiserId": "xxxxxxxxxxxxx"
},
"reportType": "conversion",
"columns": [
{
"columnName": "conversionVisitExternalClickId"
},
{
"columnName": "conversionTimestamp"
},
{
"columnName": "keywordId"
},
{
"columnName": "adGroup"
},
{
"columnName": "campaign"
},
{
"columnName": "account"
},
{
"columnName": "floodlightActivity"
},
{
"columnName": "conversionId"
}
],
"timeRange": {
"startDate": "2014-01-1",
"endDate": "2016-11-3"
},
"includeRemovedEntities": false,
"downloadFormat": "csv",
"statisticsCurrency": "agency",
"maxRowsPerFile": 6000000
},
"statisticsCurrencyCode": "USD",
"statisticsTimeZone": "America/New_York",
"rowCount": 5225201,
"files": [
{
"url": "https://www.googleapis.com/doubleclicksearch/v2/reports/AAAncMCcKrSJ9MS_/files/0",
"byteCount": "1271002762"
}
]
}
reports.getFile (Response)
GET https://www.googleapis.com/doubleclicksearch/v2/reports/AAAncMCcKrSJ9MS_/files/0?key={YOUR_API_KEY}
200
- Hide headers -
cache-control: private, max-age=0, must-revalidate, no-transform
content-length: 1694670352
Content-Type: text/csv
date: Tue, 21 Mar 2017 14:24:03 GMT
etag: W/"u6ybyji1w_vhYZMTUlOUhEWemRI/nWLUgbdAKD9EU6srfscDcZyGxz8"
expires: Tue, 21 Mar 2017 14:24:03 GMT
server: UploadServer
vary: Origin, X-Origin
x-guploader-uploadid: AEnB2UoZzz4AafhGUKg-1edKZWLCmiML_a6HnP7lr11aD7tXZAnEIgyEno9ZhHOCbJ02XTlAkgtYcD4AOGbhmar6dpSx-uzh-n_6SCQH-iShQma-k6GLsMk

416 from cloudfront when S3 origin video/mp4 resource replaced with bigger file

Why is cloudfront responding with a 416 status in the following scenario, even though I believe the requested range should be satisfiable?
Original S3 object
The scenario:
configure cloudfront distribution with an S3 origin, Object Caching set to "Customize", min/max/default TTL all set to 0
upload a video/mp4 file to S3, with no cache related headers set
playback video in chrome, via cloudfront
At this point, everything seems fine. I see 3 network requests to the resource - why chrome is requesting overlapping ranges in 2 and 3, I don't understand, but nevertheless, the video plays fine at this point.
Overwritten S3 object
Now, soon after:
upload larger (~ 2x) file to same S3 location
playback video in chrome, via cloudfront
This time, an error occurs.
It seems as though resp 1 here is ok (i.e. content length looks to be correct) - although not sure why it's a Miss from cloudfront - I would've expected it to be a RefreshHit given the TTL settings
Why is "req 2" of second page load failing with a 416, even though the requested range is within the content length from resp 1?
Notes:
going direct to S3, there is no problem
replacing the bigger file back to the smaller file, the video plays fine again
uploading the file to S3 with "cache-control: no-cache", there is no problem but I always get a "Miss from cloudfront". My understanding is that cloudfront should always be checking with S3 to see if content has changed with this configuration
waiting for some time (say 30s) - the problem resolves itself
previous attempts at replicating this scenario saw an addition 304 response from S3 to CloudFront after the overwrite, but I haven't been able to replicate that exact behaviour (there is just a single 216 now)
Logs
Original S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "535351" },
{ "name": "x-amz-cf-id", "value": "-6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=524288-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 524288-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "11063" },
{ "name": "x-amz-cf-id", "value": "_z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=32768-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 32768-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "502583" },
{ "name": "x-amz-cf-id", "value": "8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ==" }
]
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:23:53 MEL50 34771 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - Error -6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw== dvayusv1lektq.cloudfront.net https 46 1.613 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:23:56 MEL50 11408 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit _z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A== dvayusv1lektq.cloudfront.net https 47 2.194 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
2016-12-11 23:23:56 MEL50 503468 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit 8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ== dvayusv1lektq.cloudfront.net https 47 0.259 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:44 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> FB18FDABAA5DF6CA REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 535351 6448 118 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:53 +0000] 54.239.202.78 - 934DB75AC20953C3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 535351 535351 101 98 "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:54 +0000] 54.239.202.78 - 56EF7496F985D4B3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 11 - "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:56 +0000] 54.239.202.78 - 7F0087B4769D0FD3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 4 - "-" "Amazon CloudFront" -
Overwritten S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:24:07 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-956851/956852" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "956852" },
{ "name": "x-amz-cf-id", "value": "CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 416,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "range", "value": "bytes=917504-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "server", "value": "CloudFront" },
{ "name": "x-cache", "value": "Error from cloudfront" },
{ "name": "content-type", "value": "text/html" },
{ "name": "status", "value": "416" },
{ "name": "content-length", "value": "49" },
{ "name": "x-amz-cf-id", "value": "WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA==" },
{ "name": "expires", "value": "Sun, 11 Dec 2016 23:24:19 GMT" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 0,
"request_headers": [
{ "name": "User-Agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "Range", "value": "bytes=32768-" },
{ "name": "Accept-Encoding", "value": "identity;q=1, *;q=0" }
],
"response_headers": []
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:24:19 MEL50 52198 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A== dvayusv1lektq.cloudfront.net https 46 1.551 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:24:19 MEL50 302 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 416 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA== dvayusv1lektq.cloudfront.net https 47 0.001 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Error HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:06 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> E58826B8A66DBA1B REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 956852 7367 53 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:18 +0000] 54.239.202.45 - BA1C06FEA7DAC83F REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 956852 956852 40 35 "-" "Amazon CloudFront" -
What you can't see is which object -- the old one or the new one -- S3 is serving to CloudFront on the first requests after the overwrite.
When you overwrite an existing object in S3, the overwrite is always an atomic operation, in the sense that every request will be satisfied with either the complete old object or the complete new object... but the timing is not guaranteed, because S3 has an eventual consistency model for overwrites of existing objects.
Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all regions.
Updates to a single key are atomic. For example, if you PUT to an existing key, a subsequent read might return the old data or the updated data, but it will never write corrupted or partial data.
http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
Even if the bucket does not have versioning enabled, there's a window of time -- usually very short -- when both objects exist in S3. It can't be any other way, given that the above statement is true.
Similarly, it's possible to request a nonexistent object from S3, then upload the object, then request it again and continue to get a 404 (or 403, depending on bucket configuration) response for a short time before downloads succeed.
Conversely, if there has never been an attempt to download a nonexistent object, you will always find it immediately available for download after you upload it. This is a tradeoff that is essentially a design necesssity at large scale.
The bucket and distribution logs should help reveal what's happening behind the scenes, particularly if the CloudFront download from S3 on any of the download attempts after the upload shows a byte count consistent with the old object.
Upon further reflection... there is a second possible explanation what is happening, here, and it is difficult to decide whether or not this might be considered a bug in CloudFront, if indeed this is the actual issue.
After the object is replaced in S3, let's assume for a moment that consistency is not an issue -- let's assume thst all subsequent requests after the upload do in fact cause S3 to provide the current version of the object to CloudFront.
The problem here may be that the first download after the object was replaced -- which should have caused CloudFront to evict its old cached version of the object -- is being canceled by the browser. (I've seen Chrome do this, though I don't know exactly why it does it.)
When a download is canceled, the response from the origin is not cached.
Canceled Requests
If an object is not in the edge cache, and if a viewer terminates a session (for example, closes a browser) after CloudFront gets the object from your origin but before it can deliver the requested object, CloudFront does not cache the object in the edge location.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorS3Origin.html#response-s3-canceled-requests
Granted, this says "if the object is not in the edge cache," and you could argue that it actually is... but it may be a question of semantics: the object CloudFront is requesting from S3 is arguably not in the cache -- what's in the cache is a different object (at the same URI). If this logic holds, then the assertion that "CloudFront does not cache the object" presumably still holds, as well.
So... request 2 asks for a range outside of the range of the old object, which is the only thing CloudFront knows about. CloudFront dutifully reports the fact that -- for all it knows -- the request is indeed out of bounds, requested range not satisfiable.
Arguably, CloudFront should verify that the object is fresh, first... but also arguably, the browser should not make a subsequent range request for an object it has never successfully downloaded.
Maybe we can shed some light on this with correlated log files.

Resources