Interesting behavior in Media Source Extensions - ffmpeg

Question

I'm trying to build a fairly standard video player using Media Source Extensions; however, I want the user to be able to control when the player moves on to a new video segment. For example, we might see the following behavior:
Video player plays 1st segment
Source Buffer runs out of data causing the video to appear paused
When the user is ready, they click a button that adds the 2nd segment to the Source Buffer
The video continues by playing the 2nd segment
This works well, except that when the video appears paused during step 2 it doesn't stop at the last frame of the 1st segment. Instead, it stops two frames before the end of the 1st segment. Those last two frames aren't being dropped, they just get played after the user clicks the button to advance the video. This is an issue for my application, and I'm trying to figure out a way to make sure all of the frames from the 1st segment get played before the end of step 2.
I suspect that these last two frames are getting held up in the video decoder buffer. Especially since calling endOfStream() on my Media Source after adding the 1st segment to the Source Buffer causes the 1st segment to play all the way through with no frames left behind.
Additional Info
I created each video segment file from a series of PNGs using the following ffmpeg command
ffmpeg -i %04d.png -movflags frag_keyframe+empty_moov+default_base_moof video_segment.mp4
Maybe this is a clue? End of stream situations not handled correctly (last frames are dropped)
Another interesting thing to note is that if the video only has 2 frames or less, MSE doesn't play it at all.
The browser I'm using is Chrome. The code for my MSE player is just taken from the Google Developers example, but I'll post it here for completeness. This code only covers up to step 2 since that's where the issue is.
<script>
const mediaSource = new MediaSource();
video.src = URL.createObjectURL(mediaSource);
mediaSource.addEventListener('sourceopen', sourceOpen, { once: true });
function sourceOpen() {
URL.revokeObjectURL(video.src);
const sourceBuffer = mediaSource.addSourceBuffer('video/mp4; codecs="avc1.64001f"');
sourceBuffer.mode = 'sequence';
// Fetch the video and add it to the Source Buffer
fetch('https://s3.amazonaws.com/bucket_name/video_file.mp4')
.then(response => response.arrayBuffer())
.then(data => sourceBuffer.appendBuffer(data));
}

Answer 1

This works well, except that when the video appears paused during step 2 it doesn't stop at the last frame of the 1st segment. Instead, it stops two frames before the end of the 1st segment. Those last two frames aren't being dropped, they just get played after...
This behavior is browser dependent. Let's start with spec quote:
When the media element needs more data, the user agent SHOULD transition it from HAVE_ENOUGH_DATA to HAVE_FUTURE_DATA early enough for a web application to be able to respond without causing an interruption in playback. For example, transitioning when the current playback position is 500ms before the end of the buffered data gives the application roughly 500ms to append more data before playback stalls.
The behavior you are seeing is that MSE compatible browser is aware that the stream is not yet ended but it also is aware that it is running out of data. It indicates need in further data by changing its ready state however it does not have an obligation to play out every frame it already holds. It enters buffering state based on clock time of current playback vs. end of available data.
Even though link above says...
For example, in video this corresponds to the user agent having data from the current frame, but not the next frame
...actual implementations might interpret this differently and switch to HAVE_CURRENT_DATA a bit too early, that is holding a few more video frames but knowing that it is not yet end of stream and further frames are missing. It is a sort of browser implementation specificity you just have to live with.