HTTP2 push and native ES modules: "entry" module push is ignored - http2

Question

I’ve been experimenting with approaches to serving native ES modules over HTTP2. Pretty much everything works great (where supported), but there’s a quirk that I can’t make much sense of.
Given a request for the / document, I push the resources which directly or indirectly are known to be dependencies of that document. In this case that ends up being three additional resources that piggyback via pushes:
/index.css (a dependency via <link href ...>)
/index.js (a dependency via <script type="module" src ...>
/routes.js (an indirect dependency, imported by index.js)
All three resources appear to push successfully from the server side. However, Chrome makes a second request for "/index.js" despite the push with the first request. Neither of the other two resources are requested; those pushed responses appear to be acknowledged correctly.
At first I thought this was a Chrome quirk, just a rough edge on a newly minted feature. But the same behavior is demonstrated in Firefox when the module support flag is enabled, which made me wonder if this is deliberate for some reason.
Logging from backend corresponding to above requests:
RECEIVED REQUEST: GET /
...PUSHING /index.css
...PUSHING /index.js
...PUSHING /routes.js
RECEIVED REQUEST: GET /index.js
...PUSHING /routes.js
Following up on the instructions from #sbordet: here are transcripts from both requests (great to know this stuff can be introspected in Chrome!):
First Req (/)
3067: HTTP2_SESSION
death.tips:443 (DIRECT)
Start Time: 2017-10-09 10:49:24.597
t=304289 [st= 0] +HTTP2_SESSION [dt=?]
--> host = "death.tips:443"
--> proxy = "DIRECT"
t=304289 [st= 0] HTTP2_SESSION_INITIALIZED
--> protocol = "h2"
--> source_dependency = 3064 (SOCKET)
t=304289 [st= 0] HTTP2_SESSION_SEND_SETTINGS
--> settings = ["[id:1 (SETTINGS_HEADER_TABLE_SIZE) value:65536]","[id:3 (SETTINGS_MAX_CONCURRENT_STREAMS) value:1000]","[id:4 (SETTINGS_INITIAL_WINDOW_SIZE) value:6291456]"]
t=304289 [st= 0] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 15663105
--> window_size = 15728640
t=304289 [st= 0] HTTP2_SESSION_SEND_WINDOW_UPDATE
--> delta = 15663105
--> stream_id = 0
t=304289 [st= 0] HTTP2_SESSION_SEND_HEADERS
--> exclusive = true
--> fin = true
--> has_priority = true
--> :method: GET
:authority: death.tips
:scheme: https
:path: /
pragma: no-cache
cache-control: no-cache
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3236.0 Safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
--> parent_stream_id = 0
--> source_dependency = 3060 (HTTP_STREAM_JOB)
--> stream_id = 1
--> weight = 256
t=304310 [st=21] HTTP2_SESSION_RECV_SETTINGS
t=304310 [st=21] HTTP2_SESSION_SEND_SETTINGS_ACK
t=304313 [st=24] HTTP2_SESSION_RECV_SETTINGS_ACK
t=304336 [st=47] HTTP2_SESSION_RECV_PUSH_PROMISE
--> :scheme: https
:authority: death.tips
:path: /index.css
:method: GET
--> id = 1
--> promised_stream_id = 2
t=304336 [st=47] HTTP2_STREAM_SEND_PRIORITY
--> exclusive = true
--> parent_stream_id = 1
--> stream_id = 2
--> weight = 110
t=304336 [st=47] HTTP2_SESSION_RECV_PUSH_PROMISE
--> :scheme: https
:authority: death.tips
:path: /index.js
:method: GET
--> id = 1
--> promised_stream_id = 4
t=304336 [st=47] HTTP2_STREAM_SEND_PRIORITY
--> exclusive = true
--> parent_stream_id = 2
--> stream_id = 4
--> weight = 110
t=304336 [st=47] HTTP2_SESSION_RECV_PUSH_PROMISE
--> :scheme: https
:authority: death.tips
:path: /routes.js
:method: GET
--> id = 1
--> promised_stream_id = 6
t=304336 [st=47] HTTP2_STREAM_SEND_PRIORITY
--> exclusive = true
--> parent_stream_id = 4
--> stream_id = 6
--> weight = 110
t=304336 [st=47] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 388
content-type: text/html; charset=utf-8
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "c3QDLn1lTsAqsErFvMgM3bEsUsY="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 1
t=304336 [st=47] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 88
content-type: text/css
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "/qkigeCvJgEE+0+5YhHLgByhKL0="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 2
t=304336 [st=47] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 60
content-type: text/javascript
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "/+cUWoFWkafsB6vSI5wBuB7v4Tk="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 4
t=304336 [st=47] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 64
content-type: text/javascript
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "2ZM3pEXqn9z1d5tkBr2x5kdHsGk="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 6
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 388
--> stream_id = 1
t=304336 [st=47] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -388
--> window_size = 15728252
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 1
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 88
--> stream_id = 2
t=304336 [st=47] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -88
--> window_size = 15728164
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 2
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 60
--> stream_id = 4
t=304336 [st=47] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -60
--> window_size = 15728104
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 4
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 64
--> stream_id = 6
t=304336 [st=47] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -64
--> window_size = 15728040
t=304336 [st=47] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 6
t=304337 [st=48] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 388
--> window_size = 15728428
t=304342 [st=53] HTTP2_STREAM_ADOPTED_PUSH_STREAM
--> stream_id = 2
--> url = "https://death.tips/index.css"
t=304343 [st=54] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 88
--> window_size = 15728516
Second Req (/index.js)
3085: HTTP2_SESSION
death.tips:443 (DIRECT)
Start Time: 2017-10-09 10:49:24.694
t=304386 [st= 0] +HTTP2_SESSION [dt=?]
--> host = "death.tips:443"
--> proxy = "DIRECT"
t=304386 [st= 0] HTTP2_SESSION_INITIALIZED
--> protocol = "h2"
--> source_dependency = 3084 (SOCKET)
t=304386 [st= 0] HTTP2_SESSION_SEND_SETTINGS
--> settings = ["[id:1 (SETTINGS_HEADER_TABLE_SIZE) value:65536]","[id:3 (SETTINGS_MAX_CONCURRENT_STREAMS) value:1000]","[id:4 (SETTINGS_INITIAL_WINDOW_SIZE) value:6291456]"]
t=304386 [st= 0] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 15663105
--> window_size = 15728640
t=304386 [st= 0] HTTP2_SESSION_SEND_WINDOW_UPDATE
--> delta = 15663105
--> stream_id = 0
t=304386 [st= 0] HTTP2_SESSION_SEND_HEADERS
--> exclusive = true
--> fin = true
--> has_priority = true
--> :method: GET
:authority: death.tips
:scheme: https
:path: /index.js
pragma: no-cache
cache-control: no-cache
origin: https://death.tips
user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3236.0 Safari/537.36
accept: */*
referer: https://death.tips/
accept-encoding: gzip, deflate, br
accept-language: en-US,en;q=0.9
--> parent_stream_id = 0
--> source_dependency = 3080 (HTTP_STREAM_JOB)
--> stream_id = 1
--> weight = 220
t=304405 [st=19] HTTP2_SESSION_RECV_SETTINGS
t=304405 [st=19] HTTP2_SESSION_SEND_SETTINGS_ACK
t=304409 [st=23] HTTP2_SESSION_RECV_SETTINGS_ACK
t=304409 [st=23] HTTP2_SESSION_RECV_PUSH_PROMISE
--> :scheme: https
:authority: death.tips
:path: /routes.js
:method: GET
--> id = 1
--> promised_stream_id = 2
t=304409 [st=23] HTTP2_STREAM_SEND_PRIORITY
--> exclusive = true
--> parent_stream_id = 1
--> stream_id = 2
--> weight = 110
t=304409 [st=23] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 60
content-type: text/javascript
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "/+cUWoFWkafsB6vSI5wBuB7v4Tk="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 1
t=304409 [st=23] HTTP2_SESSION_RECV_HEADERS
--> fin = false
--> :status: 200
cache-control: public, max-age=0
content-encoding: deflate
content-length: 64
content-type: text/javascript
date: Mon, 09 Oct 2017 14:49:24 GMT
etag: "2ZM3pEXqn9z1d5tkBr2x5kdHsGk="
last-modified: Mon, 09 Oct 2017 14:43:24 GMT
--> stream_id = 2
t=304409 [st=23] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 60
--> stream_id = 1
t=304409 [st=23] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -60
--> window_size = 15728580
t=304409 [st=23] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 1
t=304409 [st=23] HTTP2_SESSION_RECV_DATA
--> fin = false
--> size = 64
--> stream_id = 2
t=304409 [st=23] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = -64
--> window_size = 15728516
t=304409 [st=23] HTTP2_SESSION_RECV_DATA
--> fin = true
--> size = 0
--> stream_id = 2
t=304410 [st=24] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 60
--> window_size = 15728576
t=304412 [st=26] HTTP2_STREAM_ADOPTED_PUSH_STREAM
--> stream_id = 2
--> url = "https://death.tips/routes.js"
t=304413 [st=27] HTTP2_SESSION_UPDATE_RECV_WINDOW
--> delta = 64
--> window_size = 15728640

Answer 1

This was quite a mystery!
The issue is that — well, I’m not gonna be able to explain this well, but my shallow understanding is that documents are requested "with credentials", but <script type="module"> triggers, by default, a "no credentials" request. The push promise for the script is "with credentials" by association, but never the twain shall meet. So the browser must make a new request because the push promise "doesn’t count". And there is a solution:
<script type="module" src="/index.js" crossorigin="use-credentials">
I would never have thought to use a "crossorigin" attribute to fetch a resource on the same site, but there it is. Push gets adopted, and my little experiment just got twice as fast.
Here’s the transcript of the whole conversation in #whatwg:
[7:35pm] <bathos> I’ve got a question about interactions between module
loading and HTTP2 that’s had me scratching my head for a few days — is
that something appropriate to ask about here?
[7:37pm] <jyasskin> bathos: Yes.
[7:39pm] <bathos> Cool. I’ve been experimenting with serving resources using
HTTP2 push — assemble a dep graph in advance and follow through on
requests by provisioning their known dependencies as push promises. This
works great on the whole, but there’s a quirk I’ve observed that seems to
be related specifically to ES module "entrypoints".
[7:40pm] <bathos> I asked about it on SO, so there’s a bit of detail in the
question and comments there: https://stackoverflow.com/questions/46642569/http2-push-and-native-es-modules-entry-module-push-is-ignored
[7:40pm] <bathos> The gist though:
[7:41pm] <bathos> Given a request for a document which contains
<script type="module" src="something">, and an http2 session which
includes a push promise for "something", the "something" push is never
adopted. Instead, the browser makes a fresh request for it.
[7:41pm] <jyasskin> Domenic: ^
[7:42pm] <bathos> Dependencies imported _in_ ES are adopted.
[7:42pm] <jyasskin> bathos: I'm not an expert here, but your question
reminds me of the with-vs-no-credentials problem in
https://github.com/whatwg/fetch/issues/354.
[7:42pm] <bathos> And if I reference the same module in a different way in
the root document, e.g. a preload <link>, it is successfully adopted. It’s
peculiar to type="module"
[7:43pm] <bathos> oh, interesting
[7:43pm] <jyasskin> Apologies if I've just sent you on a wild goose chase.
[7:44pm] <bathos> I have been on a lot for the last two days haha! Since
HTTP2 is still pretty mysterious to me, it’d been hard to rule out the
possibility that I’m doing something weird there, though I’m pretty sure
at this point that I’m not.
[7:52pm] <bathos> jyasskin you genius!
[7:53pm] <jyasskin> s/genius/pattern-matcher/ :)
[7:53pm] <bathos> crossorigin="use-credentials" in the doc actually makes
the module push promise get adopted
[7:54pm] <bathos> I never would have thought to try "crossorigin" on a file
on the same host haha

Answer 2

The browser activity is the following:
send request (stream=1)
receive push promise for /index.css (stream=2)
receive push promise for /index.js (stream=4)
receive push promise for /routes.js (stream=6)
receive headers for stream=1
receive headers for stream=2
receive headers for stream=4
receive headers for stream=6
receive data for stream=1 (388+0 bytes)
receive data for stream=2 (88+0 bytes)
receive data for stream=4 (60+0 bytes)
receive data for stream=6 (64+0 bytes)
My interpretation is that the browser receives the whole body for the primary request (stream=1) before it receives the whole body for the pushed resources.
I'm guess internally the browser starts parsing the HTML, figure out it needs /index.js, find that it is not yet arrived although it has been promised, and therefore it issues a request for it.
The browser probably needs /index.css later than it needs /index.js, and by the time it needs the CSS it has already arrived to the browser as a pushed resource, and that would explain why /index.css is used from the push cache.
If you can control when the resources are written to the network, try to send the whole body of /index.js before sending the HTML body. That should make the browser aware that index.js is fully available in the push cache and use it from there, rather than requesting it anew.
A final note that the push cache implementation in Chrome has varied greatly over the years/months, so what could be true today may not hold in the future.