Reference existing WebGL depth buffer when rendering a new ThreeJS scene - three.js

I have an existing WebGL canvas that is being rendered without using ThreeJS, and is for all intents and purposes a black box to me, apart from two facts: (1) I have access to the underlying webgl canvas DOM element and can position and resize it on the screen, and (2) I know the properties of the camera for the scene, and get updates on every render cycle for that camera.
The problem I need to solve can be simplified to the following: I need to have my own separate ThreeJS canvas that displays both the black box canvas data, and then elements that I draw, like a cube for a simple example. I can already easily overlay the two canvases, set the transparency on my canvas for everything but the cube, and then align the two with the camera events from the black box library. This works quite well.
The issue with this is that when I draw my objects, like a cube, they don't respect the depth buffer of the black box canvas. So I might have a cube that is properly aligned with the backing scene and movements of the scene, but then it isn't properly masked when something in the black box canvas is closer to the camera than the cube. My thought is that I need to solve this in one of two ways: (1) I can have my renderer write to the other canvas with autoClear = false and preserveDrawingBuffer = true, or (2) I can somehow copy the depth buffer from the black box canvas into my canvas, and then set up my renderer so that it respects the new depth buffer.
I haven't been successful with either approach yet, so I'm wondering if this is possible, and if so which of the above approaches, or what other approach, can solve this problem?
--Edit--
See https://jsfiddle.net/zdxyoajb/ for angular/typescript implementation of the above attempts. In the following animate loop, if I comment out the overlayRenderer lines, the below sphere will be red and offset from the center (as it should be), but if I don't comment the lines, I get the below image. I also get the following error:
WebGL: INVALID_OPERATION: uniformMatrix4fv: location is not from current program
animate() {
requestAnimationFrame(() => this.animate());
this.blackBoxCamera.copy(this.overlayCamera);
this.blackBoxRenderer.render(this.blackBoxScene, this.blackBoxCamera);
this.overlayRenderer.state.reset();
this.overlayRenderer.render(this.overlayScene, this.overlayCamera);
}

Related

User painting on a canvas within A-Frame

I have an A-Frame scene that contains, among others, a <canvas> element that is the material source for a 3D scene object. I can paint on the canvas programmatically, and it shows up as texture. So far, so good.
However, I'd now also like to enable the user to paint something on the canvas using the controllers. I have added two raycasters/controls:
<a-entity laser-controls="hand: left" raycaster="objects: table2"></a-entity>
<a-entity laser-controls="hand: right" raycaster="objects: table2"></a-entity>
And on the table2 object, I have added a raycaster-listen mixin as described in https://aframe.io/docs/1.3.0/components/raycaster.html#listening-for-raycaster-intersection-data-change.
This works in so far as I get the console log entries with the world coordinates of the intersection point, but I'm absolutely stuck at how to get from the world coordinates back to the canvas coordinates I need to actually paint in the right spot.
In addition, it seems no canvas draw commands I issue in the raycaster-listen tick callback actually have any visible effect (regardless of coordinates).
Any hints appreciated!
As usual, I figured it out the next day 😉
[...] I'm absolutely stuck at how to get from the world coordinates back to the canvas coordinates I need to actually paint in the right spot.
Solution found at https://discourse.threejs.org/t/convert-camera-frustrum-to-uv-coordinate-on-texture/16791/2 - just use intersection.uv which actually contains the normalized texture coordinates of the intersection point. Scale by canvas width/height and you're done.
[...] it seems no canvas draw commands I issue in the raycaster-listen tick callback actually have any visible effect.
Solution found at aframe not rendering lottie json texture mapped to canvas but works in three.js - set texture.needsUpdate = true; in the tick callback after drawing on the canvas.

Rendering Sprites in Three.js over 3D Object and also behind without clipping

I need a way to render a sprite always over top of a 3DObject, but it also must disappear behind it when the camera is rotated.
I first tried disabling the depth-buffer write but this will always render the Sprite in front.
Is there a way to hide it when its behind, but prevent clipping with the Object?
Edit 1:
The Problem is not exactly about z-fighting because the depthbuffer values of the sprite and the object are not equal.
The Sprite is a Billboard, so it will always face the camera, but it clips with the object, when rotating the camera.
Thank you for your help

Black background for three.js sprites with depthTest true

I have been experimenting with converting the custom attribute BufferGeometry example (https://threejs.org/examples/webgl_buffergeometry_custom_attributes_particles.html) into a fly through animation and I find that the sprites have dark backgrounds (and those that I create myself as well) if depthTest is set to true. See the image.
The sprite in the custom attribute example has a transparent background but this appears to be ignored when it is rendered if depthTest is set to true.
I have tried numerous custom blending rules but cannot find a way to remove the background, only to reduce the effect a bit. The background disappears if depthTest is set to false.
Is this a known limitation? Is there a work around?
I am modifying this question to add clearer images with a different ball sprite (also with a transparent background). This image has depthTest set to true for the custom ShaderMaterial used in the https://threejs.org/examples/webgl_buffergeometry_custom_attributes_particles.html three.js example.
By comparison, this uses multiple PointsMaterials from a different three.js example (https://threejs.org/examples/webgl_points_sprites.html), also with depthTest set to true and using the PointsMaterial map parameter for the sprite.
As you can see, the second PointsMaterial example works as expected. Because PointsMaterial only accepts one fixed size and color, I need to create 36 different point geometries to render this image.
I would prefer to use the custom shader in the first example (which has custom size and color attributes and requires only one geometry). Is there a way to define a custom shader to support depthTest like the PointsMaterial does?
WebGL has a depth buffer that's used to find out which pixels are hiding behind other pixels. If a pixel is hiding behind another, it doesn't get rendered. Typically that's what you would want, why would you waste GPU time rendering pixels you're never going to see? Consider the example depthbuffer below, white squares are closer to the camera, while darker squares are further away:
If there was a square hiding behind another one, you wouldn't see it in the depthBuffer, you'd only see the closest one. The depthBuffer doesn't know that your squares have partial transparency and you want some parts to be see-through. This is an issue in other realtime engines too, this is an example from Unity:
The squares that are closer are blocking part of the squares that are further away in the depthBuffer, so the occluded pixels don't get rendered. That's why setting depthTest to false is useful. You're basically telling the renderer to stop testing which pixels are behind others, and just render all of them. Here it is again with depthTest = false:
If you must set depthTest on, (for instance, if you want the sprites to hide behind a solid wall) you could set depthWrite to false on the sprite material. It would still check which pixels are behind others, but the pixels from the sprites wouldn't get written to the depthBuffer, so no sprite could block another sprite.
depthTest: true; // Tests pixel depth
depthTest: false; // Does not test pixel depth
depthWrite: true; // Writes pixel depth to depthBuffer
depthWrite: false; // Does not write pixel depth to depthBuffer
As far as blending mode, I prefer THREE.AdditiveBlending because it gives the particles a nice glowing effect when they accumulate one on top of the other. But that's up to you.

Three.js - DoubleSided Semi-Transparent Materials Only Work From Certain Angles

I am currently using three.js and trying to create a 3D experience with a semi-transparent material that can be viewed from all angles. I've noticed that depending on the camera angle, only certain portions of the mesh are semi-transparent and will show the content behind them. In this example below I've created two half cylinders and applied the same transparent material with the stack overflow logo. The half cylinder on the left properly shows the logo on the closest surface, as well as the surface behind it. The half cylinder on the right only shows the logo on the closest surface and fails to render the logo that wraps behind it. However, it does properly render the background image so the material is still treated correctly as transparent:
If I spin the orbital camera around 180 degrees the side that originally failed to see through now works and the other side exhibits the wrong behavior. This leads me to believe it's related to the camera position / depth sorting. The material in this case is a standard MeshPhongMaterial with transparent set to true, side as DoubleSide, and a single map for the transparent stack overflow logo. The geometry is formed from an opened ended CylinderGeometry. Any help would be greatly appreciated!

Creating a magnifying-glass effect in three.js WebGL

I'm working with an orthographic view in three.js/WebGL renderer, and I want a magnifying glass that tracks with the user mouse. I'm looking for the best way of doing this that's efficient.
When working with html5 canvas raw commands, this was easy: I simply defined a circular clip region, zoomed my coordinates, and re-drew the whole scene. With 3d objects, it's less obvious how do to it.
The method I've found so far is to do the following:
Define a second camera that looks into the zoomed region. Set the orthographic clip coordinates to be small so that it doesn't need to do much work
Create a THREE.WebGLRenderTarget
Tell the renderer and all my line textures that the resolution is about to change
Render the scene into the RenderTarget
Add a CircleGeometry as a MeshObject at the spot at the mouse position (in world coordinate but above the rest of the scene, close to the camera). Call this the lens.
Give the lens the WebGLRenderTarget as a texture.
Go back to my default camera, reset all my resolution parameters, and redraw the scene with the 'lens' object added.
This works (see image below) but I'm worried about parts of it:
I have to render twice per frame
Lines don't draw well, because the resolution problems. I have to keep track of all materials that need to know screen resolution and update all of them twice per screen render.
Related problems:
I want to overlay some plot axes on top of this, and possibly gridlines. These would change as the view pans. I'm not sure if I should make these 3d objects, or do it in a 2d canvas context I lay overtop.
I want to overlay some plot lines, and have them show up sensibly in the zoomed view. "Sensible" here is hard to figure out: I don't want them too fat in the zoomed view, but I also don't want to scale them up as much as the image detail (which is being rendered as a texture onto Plane objects behind).
This is a long post, but I'm still new to three.js and looking for good ideas.

Resources