How to mix / combine multiple WebRTC media streams (screen capture + webcam) into a single stream? - html5-canvas

Question

I have a live screen capture media stream returned from getDisplayMedia(),
and a live webcam media stream returned from getUserMedia().
I currently render the webcam video on top of the screen share video, to create a picture-in-picture effect:
I want to mix / combine them into one video stream, in order to render it inside a single HTML video element.
I want to keep both streams active & live just as if they were two separate videos rendering two different media streams.
I also need to maintain the streams positions - keeping the webcam stream small and on top of the screen share stream. It's also really important for me to keep the original resolution and bitrate.
How can I do that?

Answer 1

You can draw both video streams on an HTMLCanvasElement, and then create a MediaStream from this HTMLCanvasElement calling its captureStream method.
To draw the two video streams, you'd have to read them in <video> elements and drawImage() these <video> elements onto the canvas in a timed loop (e.g requestAnimationFrame can be used for this timed loop).
async function getOverlayedVideoStreams( stream1, stream2 ) {
// prepare both players
const vid1 = document.createElement("video");
const vid2 = document.createElement("video");
vid1.muted = vid2.muted = true;
vid1.srcObject = stream1;
vid2.srcObject = stream2;
await Promise.all( [
vid1.play(),
vid2.play()
] );
// craete the renderer
const canvas = document.createElement("canvas");
let w = canvas.width = vid1.videoWidth;
let h = canvas.height = vid1.videoHeight;
const ctx = canvas.getContext("2d");
// MediaStreams can change size while streaming, so we need to handle it
vid1.onresize = (evt) => {
w = canvas.width = vid1.videoWidth;
h = canvas.height = vid1.videoHeight;
};
// start the animation loop
anim();
return canvas.captureStream();
function anim() {
// draw bg video
ctx.drawImage( vid1, 0, 0 );
// caculate size and position of small corner-vid (you may change it as you like)
const cam_w = vid2.videoWidth;
const cam_h = vid2.videoHeight;
const cam_ratio = cam_w / cam_h;
const out_h = h / 3;
const out_w = out_h * cam_ratio;
ctx.drawImage( vid2, w - out_w, h - out_h, out_w, out_h );
// do the same thing again at next screen paint
requestAnimationFrame( anim );
}
}
Live demo as a glitch since StackSnippets won't allow capture APIs.

Answer 2

I tried rendering on canvas but the refresh rate drops whenever i change/minimize my web-app tab. I was able to fix that using audioTimerLoop.
But it still did'nt work on firefox.
As i was limited to chrome now, i used PictureInPicture api to display the userMedia as a PiP and then just recorded the screenMedia.
This let the user to adjust their cam video coordinates which were fixed in the canvas method.
P.S. For getting pip mode.I displayed the userMedia onscreen and set its opacity to 0% and created a button to toggle it.