I have this function which checks an input data structure that it has the correct fields.
def validate_response(RequiredTypes: RawCustomResponseDataStruct, ResultToCheck: RawCustomResponseData) -> bool:
if isinstance(RequiredTypes, dict) and isinstance(ResultToCheck, dict):
return all(k in ResultToCheck and validate_response(RequiredTypes[k], ResultToCheck[k]) for k in RequiredTypes)
if isinstance(RequiredTypes, list) and isinstance(ResultToCheck, list):
return all(validate_response(RequiredTypes[0], c) for c in ResultToCheck)
if isinstance(RequiredTypes, type):
return isinstance(ResultToCheck, RequiredTypes)
return False
The class and it's fields look like this: (I've made the dictionary replica so that isinstance can be used as I don't want to import more libraries).
class Crawler(TypedDict):
is_crawler: bool
category: Union[str, None]
last_seen: Union[str, None]
CrawlerStruct = {
"is_crawler": bool,
"category": Union[str, None],
"last_seen": Union[str, None]}
When I run this test with the following input:
def test_validate_response():
ResultToCheck = {"ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36 isBot3", "type": "browser", "brand": "Apple", "name": "Mac", "url": "https://about.google/", "os": {"name": "macOS 10.14 Mojave", "code": "macos_10_14", "url": "https://en.wikipedia.org/wiki/MacOS_Mojave", "family": "macOS", "family_code": "macos", "family_vendor": "Apple Inc.", "icon": "https://assets.userstack.com/icon/os/macosx.png", "icon_large": "https://assets.userstack.com/icon/os/macosx_big.png"}, "device": {"is_mobile_device": False, "type": "desktop", "brand": "Apple", "brand_code": "apple", "brand_url": "https://www.apple.com/", "name": "Mac"}, "browser": {"name": "Chrome", "version": "71.0.3578.98", "version_major": "71", "engine": "WebKit/Blink"}, "crawler": {"is_crawler": False, "category": None, "last_seen": None}}
result = validate_response(RawCustomResponseDataStruct, user_agent)
assert result is True
It fails.
However if I remove the union, and just use str for the class and dictionary and run the test changing the "crawler" object to have the following values
"crawler": {"is_crawler": False, "category": "None", "last_seen": "None"}
It passes. Why is this? Why is it not recognising the None type?
I try to implement NLog into my .NET Core Api web service.
I want to log to an Oracle database. All works well through an nlog.config XML file.
But the goal is to implement NLog config into appsettings.json and here problem occurs.
I get the error set in title:
LayoutRenderer cannot be found: 'aspnet-user-identity
My config file is like this :
"NLog": {
"autoReload": true,
"throwConfigExceptions": true,
"internalLogLevel": "info",
"internalLogFile": "c:/app/log/dev/internal-appsetting-nlog.txt",
"extensions": {
"NLog.Extensions.Logging": {
"assembly": [
"NLog.Extensions.Logging",
"NLog.Web.AspNetCore"
]
}
},
"variables": {
"var_logdir": "c:/app/log/dev"
},
"default-wrapper": {
"type": "AsyncWrapper",
"overflowAction": "Block"
},
"targets": {
"all-file": {
"type": "File",
"fileName": "${var_logdir}/nlog-all-${shortdate}.log",
"layout": {
"type": "JsonLayout",
"Attributes": [
{
"name": "timestamp",
"layout": "${date:format=o}"
},
{
"name": "level",
"layout": "${level}"
},
{
"name": "logger",
"layout": "${logger}"
},
{
"name": "message",
"layout": "${message:raw=true}"
},
{
"name": "properties",
"encode": false,
"layout": {
"type": "JsonLayout",
"includeallproperties": "true"
}
}
]
}
},
"db": {
"type": "Database",
"commandText": "INSERT INTO logtable (LOGLEVEL,LOGGER,MESSAGE,MACHINENAME,USERNAME,CALLSITE, THREADID,EXCEPTIONMESSAGE,STACKTRACE,SESSIONID) VALUES (:pLEVEL,:pLOGGER,:pMESSAGE,:pMACHINENAME, :pCALLSITE,:pTHREADID,:pEXCEPTIONMESSAGE,:pSTACKTRACE)",
"parameters": [
{
"name": "#pLEVEL",
"layout": "${level}"
},
{
"name": "#pLOGGER",
"layout": "${logger}"
},
{
"name": "#pMESSAGE",
"layout": "${message}"
},
{
"name": "#pMACHINENAME",
"layout": "${machinename}"
},
{
"name": "#pUSERNAME",
"layout": "${aspnet-user-identity}"
},
{
"name": "#pCALLSITE",
"layout": "${callsite:filename=true}"
},
{
"name": "#pTHREADID",
"layout": "${threadid}"
},
{
"name": "#pEXCEPTIONMESSAGE",
"layout": "${exception}"
},
{
"name": "#pSTACKTRACE",
"layout": "${stacktrace}"
},
{
"name": "#pSESSIONID",
"layout": "${aspnet-sessionid}"
}
],
"dbProvider": "Oracle.ManagedDataAccess.Client.OracleConnection, Oracle.ManagedDataAccess",
"connectionString": "xxxxxxxxxxxx"
}
},
"rules": [
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "all-file"
},
{
"logger": "*",
"minLevel": "Trace",
"writeTo": "db"
},
{
"logger": "Microsoft.*",
"maxLevel": "Info",
"final": true
}
]
},
The internal debugger reports:
2019-10-09 16:48:48.6665 Info Adding target AsyncTargetWrapper(Name=all-file)
2019-10-09 16:48:48.7859 Warn Error when setting property 'Layout' on 'NLog.Targets.DatabaseParameterInfo' Exception: System.ArgumentException: LayoutRenderer cannot be found: 'aspnet-user-identity'. Is NLog.Web not included?
at NLog.Config.Factory`2.CreateInstance(String itemName)
at NLog.Layouts.LayoutParser.GetLayoutRenderer(ConfigurationItemFactory configurationItemFactory, String name)
at NLog.Layouts.LayoutParser.ParseLayoutRenderer(ConfigurationItemFactory configurationItemFactory, SimpleStringReader stringReader)
at NLog.Layouts.LayoutParser.CompileLayout(ConfigurationItemFactory configurationItemFactory, SimpleStringReader sr, Boolean isNested, String& text)
at NLog.Layouts.SimpleLayout.set_Text(String value)
at NLog.Internal.PropertyHelper.TryNLogSpecificConversion(Type propertyType, String value, Object& newValue, ConfigurationItemFactory configurationItemFactory)
at NLog.Internal.PropertyHelper.SetPropertyFromString(Object obj, String propertyName, String value, ConfigurationItemFactory configurationItemFactory)
Error occurs on ${aspnet-sessionid}. If I comment out both layout, everything works well.
I found different things on GitHub issue report but all I tried was a fail.
Could someone help?
The unknown aspnet-user-identity is probably an issue with your extensions:
"extensions": [
{ "assembly": "NLog.Extensions.Logging" },
{ "assembly": "NLog.Web.AspNetCore" }
],
Could you try the above suggestion?
P.S. Updated the wiki to include example of multiple "extensions"
Why is cloudfront responding with a 416 status in the following scenario, even though I believe the requested range should be satisfiable?
Original S3 object
The scenario:
configure cloudfront distribution with an S3 origin, Object Caching set to "Customize", min/max/default TTL all set to 0
upload a video/mp4 file to S3, with no cache related headers set
playback video in chrome, via cloudfront
At this point, everything seems fine. I see 3 network requests to the resource - why chrome is requesting overlapping ranges in 2 and 3, I don't understand, but nevertheless, the video plays fine at this point.
Overwritten S3 object
Now, soon after:
upload larger (~ 2x) file to same S3 location
playback video in chrome, via cloudfront
This time, an error occurs.
It seems as though resp 1 here is ok (i.e. content length looks to be correct) - although not sure why it's a Miss from cloudfront - I would've expected it to be a RefreshHit given the TTL settings
Why is "req 2" of second page load failing with a 416, even though the requested range is within the content length from resp 1?
Notes:
going direct to S3, there is no problem
replacing the bigger file back to the smaller file, the video plays fine again
uploading the file to S3 with "cache-control: no-cache", there is no problem but I always get a "Miss from cloudfront". My understanding is that cloudfront should always be checking with S3 to see if content has changed with this configuration
waiting for some time (say 30s) - the problem resolves itself
previous attempts at replicating this scenario saw an addition 304 response from S3 to CloudFront after the overwrite, but I haven't been able to replicate that exact behaviour (there is just a single 216 now)
Logs
Original S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "535351" },
{ "name": "x-amz-cf-id", "value": "-6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=524288-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 524288-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "11063" },
{ "name": "x-amz-cf-id", "value": "_z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498631683",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498631683" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "range", "value": "bytes=32768-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:23:54 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:23:45 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"4ab3cf8dcd7747d45c1723eb19c0c7fa\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "RefreshHit from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 32768-535350/535351" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "502583" },
{ "name": "x-amz-cf-id", "value": "8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ==" }
]
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:23:53 MEL50 34771 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - Error -6zzzNwipKKtO_L-vU3o4dbH30cBHV2zu-28rZXwVrZm5uI8oKADYw== dvayusv1lektq.cloudfront.net https 46 1.613 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:23:56 MEL50 11408 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit _z3F_A7pVXHz5PBulj8-4OeRolEzWdgT9R4-JdvgUpTLq463MZ-C_A== dvayusv1lektq.cloudfront.net https 47 2.194 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
2016-12-11 23:23:56 MEL50 503468 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498631683 - RefreshHit 8MGICqcddKwl5HZ2sNN6fpTSwO1I8qkvvurVfbBftlikXKdi-FQhdQ== dvayusv1lektq.cloudfront.net https 47 0.259 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 RefreshHit HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:44 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> FB18FDABAA5DF6CA REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 535351 6448 118 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:53 +0000] 54.239.202.78 - 934DB75AC20953C3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 535351 535351 101 98 "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:54 +0000] 54.239.202.78 - 56EF7496F985D4B3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 11 - "-" "Amazon CloudFront" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:23:56 +0000] 54.239.202.78 - 7F0087B4769D0FD3 REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 304 - - 535351 4 - "-" "Amazon CloudFront" -
Overwritten S3 object
Chrome
[
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 206,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "range", "value": "bytes=0-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "last-modified", "value": "Sun, 11 Dec 2016 23:24:07 GMT" },
{ "name": "server", "value": "AmazonS3" },
{ "name": "etag", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "status", "value": "206" },
{ "name": "x-cache", "value": "Miss from cloudfront" },
{ "name": "content-type", "value": "video/mp4" },
{ "name": "content-range", "value": "bytes 0-956851/956852" },
{ "name": "accept-ranges", "value": "bytes" },
{ "name": "content-length", "value": "956852" },
{ "name": "x-amz-cf-id", "value": "CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A==" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 416,
"request_headers": [
{ "name": ":path", "value": "/video.mp4?u=1481498656967" },
{ "name": "pragma", "value": "no-cache" },
{ "name": "accept-encoding", "value": "identity;q=1, *;q=0" },
{ "name": "accept-language", "value": "en-US,en;q=0.8,fr;q=0.6,id;q=0.4" },
{ "name": "user-agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "accept", "value": "*/*" },
{ "name": "cache-control", "value": "no-cache" },
{ "name": ":authority", "value": "dvayusv1lektq.cloudfront.net" },
{ "name": ":scheme", "value": "https" },
{ "name": "if-match", "value": "\"d4d5776a96931962b41476857f34ab6d\"" },
{ "name": "range", "value": "bytes=917504-" },
{ "name": ":method", "value": "GET" }
],
"response_headers": [
{ "name": "date", "value": "Sun, 11 Dec 2016 23:24:19 GMT" },
{ "name": "via", "value": "1.1 0ea9662a9e73b2ca5836ede6924f81b0.cloudfront.net (CloudFront)" },
{ "name": "server", "value": "CloudFront" },
{ "name": "x-cache", "value": "Error from cloudfront" },
{ "name": "content-type", "value": "text/html" },
{ "name": "status", "value": "416" },
{ "name": "content-length", "value": "49" },
{ "name": "x-amz-cf-id", "value": "WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA==" },
{ "name": "expires", "value": "Sun, 11 Dec 2016 23:24:19 GMT" }
]
},
{
"url": "https://dvayusv1lektq.cloudfront.net/video.mp4?u=1481498656967",
"response_status": 0,
"request_headers": [
{ "name": "User-Agent", "value": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36" },
{ "name": "Range", "value": "bytes=32768-" },
{ "name": "Accept-Encoding", "value": "identity;q=1, *;q=0" }
],
"response_headers": []
}
]
Cloudfront
#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version
2016-12-11 23:24:19 MEL50 52198 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 206 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error CjmlHAFcyEWCiV68Q0G3gltuQSV7maR5bUoX0CfngDgDBp5fDvI38A== dvayusv1lektq.cloudfront.net https 46 1.551 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss HTTP/2.0
2016-12-11 23:24:19 MEL50 302 150.101.108.33 GET dvayusv1lektq.cloudfront.net /video.mp4 416 - Mozilla/5.0%2520(Macintosh;%2520Intel%2520Mac%2520OS%2520X%252010_12_1)%2520AppleWebKit/537.36%2520(KHTML,%2520like%2520Gecko)%2520Chrome/55.0.2883.87%2520Safari/537.36 u=1481498656967 - Error WIb50z_8rXTdqaC4CzUSYSL0kuIE9CWlCnKNgzps7AoCSRoJplBBbA== dvayusv1lektq.cloudfront.net https 47 0.001 - TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Error HTTP/2.0
S3
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:06 +0000] 150.101.108.33 arn:aws:iam::<IAM ID>:user/<username> E58826B8A66DBA1B REST.PUT.OBJECT video.mp4 "PUT /video.mp4 HTTP/1.1" 200 - - 956852 7367 53 "-" "aws-cli/1.9.11 Python/2.7.10 Darwin/16.1.0 botocore/1.4.26" -
8cd9b72de42431df1df4dadadab73aabf29ea0b34b5d821565fe4a16a3080509 bad-video-test [11/Dec/2016:23:24:18 +0000] 54.239.202.45 - BA1C06FEA7DAC83F REST.GET.OBJECT video.mp4 "GET /video.mp4 HTTP/1.1" 206 - 956852 956852 40 35 "-" "Amazon CloudFront" -
What you can't see is which object -- the old one or the new one -- S3 is serving to CloudFront on the first requests after the overwrite.
When you overwrite an existing object in S3, the overwrite is always an atomic operation, in the sense that every request will be satisfied with either the complete old object or the complete new object... but the timing is not guaranteed, because S3 has an eventual consistency model for overwrites of existing objects.
Amazon S3 offers eventual consistency for overwrite PUTS and DELETES in all regions.
Updates to a single key are atomic. For example, if you PUT to an existing key, a subsequent read might return the old data or the updated data, but it will never write corrupted or partial data.
http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html#ConsistencyModel
Even if the bucket does not have versioning enabled, there's a window of time -- usually very short -- when both objects exist in S3. It can't be any other way, given that the above statement is true.
Similarly, it's possible to request a nonexistent object from S3, then upload the object, then request it again and continue to get a 404 (or 403, depending on bucket configuration) response for a short time before downloads succeed.
Conversely, if there has never been an attempt to download a nonexistent object, you will always find it immediately available for download after you upload it. This is a tradeoff that is essentially a design necesssity at large scale.
The bucket and distribution logs should help reveal what's happening behind the scenes, particularly if the CloudFront download from S3 on any of the download attempts after the upload shows a byte count consistent with the old object.
Upon further reflection... there is a second possible explanation what is happening, here, and it is difficult to decide whether or not this might be considered a bug in CloudFront, if indeed this is the actual issue.
After the object is replaced in S3, let's assume for a moment that consistency is not an issue -- let's assume thst all subsequent requests after the upload do in fact cause S3 to provide the current version of the object to CloudFront.
The problem here may be that the first download after the object was replaced -- which should have caused CloudFront to evict its old cached version of the object -- is being canceled by the browser. (I've seen Chrome do this, though I don't know exactly why it does it.)
When a download is canceled, the response from the origin is not cached.
Canceled Requests
If an object is not in the edge cache, and if a viewer terminates a session (for example, closes a browser) after CloudFront gets the object from your origin but before it can deliver the requested object, CloudFront does not cache the object in the edge location.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RequestAndResponseBehaviorS3Origin.html#response-s3-canceled-requests
Granted, this says "if the object is not in the edge cache," and you could argue that it actually is... but it may be a question of semantics: the object CloudFront is requesting from S3 is arguably not in the cache -- what's in the cache is a different object (at the same URI). If this logic holds, then the assertion that "CloudFront does not cache the object" presumably still holds, as well.
So... request 2 asks for a range outside of the range of the old object, which is the only thing CloudFront knows about. CloudFront dutifully reports the fact that -- for all it knows -- the request is indeed out of bounds, requested range not satisfiable.
Arguably, CloudFront should verify that the object is fresh, first... but also arguably, the browser should not make a subsequent range request for an object it has never successfully downloaded.
Maybe we can shed some light on this with correlated log files.