Does webgl drawArray() empty/discard the buffers? - performance

Trying to speed up the display of many near-identical objects in WebGL, I tried (naively, I guess), to re-use the buffers content. In the drawing routine of each object, I have (somewhat simplified):
if (! dataBuffered) {
dataBuffered = true;
:
: gl stuff here: texture loading, buffer binding and filling
:
}
// set projection and model-view matrices
gl.uniformMatrix4fv (shaderProgram.uPMatrix, false, pMatrix);
gl.uniformMatrix4fv (shaderProgram.uMVMatrix, false, mvMatrix);
// draw rectangle filled with texture
gl.drawArrays(gl.TRIANGLE_STRIP, 0, starVertexPositionBuffer.numItems);
My idea was that the texture, vertex, and texture coordinates buffer are the same, but the model-view matrix changes (same object in different places). But, alas, nothing shows up. When I comment the dataBuffered = true, it's visible.
So my question is, does drawArray() discard or empty the buffers? What else is happening? (I'm working along the lessons at learningwebgl.com, if that matters.)

Short answer is, Yes, you can reuse all the state you've set up for more than one gl.drawArrays().
http://omino.com/experiments/webgl/simplestWebGlReuseBuffers.html is a little example where it just changes one uniform float (Y-scale) and redraws, twice for every tick.
(In this case there's no textures, but some other state stays sticky.)
Hope that helps!
uniformSetFloat(gl,prog,"scaleY",1.0);
gl.drawArrays(gl.TRIANGLES, 0, posPoints.length / 3);
uniformSetFloat(gl,prog,"scaleY",0.2);
gl.drawArrays(gl.TRIANGLES, 0, posPoints.length / 3);

Related

Improving the performance of Webgl2 texSubImage2D call with large texture

Using WebGL2 I stream a 4K by 2K stereoscopic video as a texture onto the inside of a sphere in order to provide 360° VR video playback capability. I've optimized as much of the codebase as is feasible given the returns on time and the application runs flawlessly when using an .H264 video source.
However; when using 8bit VP8 or VP9 (which offer superior fidelity and file size, AV1 isn't available to me) I encounter FPS drops on weaker systems due to the extra CPU requirements for decoding VP8/VP9 video.
When profiling the app, I've identified that the per-frame call of texSubImage2D that updates the texture from the video consumes the large majority of each frame (texImage2D was even worse due to it's allocations), but am unsure how to further optimize it's use. Below are the things I'm already doing to minimize it's impact:
I cache the texture's memory space at initial load using texStorage2D to keep it as contiguous as possible.
let glTexture = gl.createTexture();
let pixelData = new Uint8Array(4096*2048*3);
pixelData.fill(255);
gl.bindTexture(GL.TEXTURE_2D, glTexture);
gl.texStorage2D(GL.TEXTURE_2D, 1, GL.RGB8, 4096, 2048);
gl.texSubImage2D(GL.TEXTURE_2D, 0, 0, 0, 4096, 2048, GL.RGB, GL.RGB, pixelData);
gl.generateMipmap(GL.TEXTURE_2D);
Then, during my render loop, both left and right eye-poses are processed for each object before moving on to the next object. This allows me to only need to call gl.bindTexture and gl.texSubImage2D once per object per frame. Additionally I also, skip populating shader program defines if the material for this entity is the same as the one for the previous entity, the video is paused, or still loading.
/* Main Render Loop Extract */
//Called each frame after pre-sorting entities
function DrawScene(glLayer, pose, scene){
//Entities are pre-sorted for transparency blending, rendering opaque first, and transparent second.
for (let ii = 0; ii < _opaqueEntities.length; ii++){
//Only render if entity and it's parent chain are active
if(_opaqueEntities[ii] && _opaqueEntities[ii].isActiveHeirachy){
for (let i = 0; i < pose.views.length; i++) {
_RenderEntityView(pose, i, _opaqueEntities[ii]);
}
}
}
for (let ii = 0; ii < _transparentEntities.length; ii++) {
//Only render if entity and it's parent chain are active
if(_transparentEntities[ii] && _transparentEntities[ii].isActiveHeirachy){
for (let i = 0; i < pose.views.length; i++) {
_RenderEntityView(pose, i, _transparentEntities[ii]);
}
}
}
}
let _programData;
function _RenderEntityView(pose, viewIdx, entity){
//Calculates/manipualtes view matrix for entity for this view. (<0.1ms)
//...
//Store reference to make stack overflow lines shorter :-)
_programData = entity.material.shaderProgram;
_BindEntityBuffers(entity, _programData);//The buffers Thomas, mind the BUFFERS!!!
gl.uniformMatrix4fv(
_programData.uniformData.uProjectionMatrix,
false,
_view.projectionMatrix
);
gl.uniformMatrix4fv(
_programData.uniformData.uModelViewMatrix,
false,
_modelViewMatrix
);
//Render all triangles that make up the object.
gl.drawElements(GL.TRIANGLES, entity.tris.length, GL.UNSIGNED_SHORT, 0);
}
let _attrName;
let _attrLoc;
let textureData;
function _BindEntityBuffers(entity, programData){
gl.useProgram(programData.program);
//Binds pre-defined shader atributes on an as needed basis
for(_attrName in programData.attributeData){
_attrLoc = programData.attributeData[_attrName];
//Bind only if exists in shader
if(_attrLoc.key >= 0){
_BindShaderAttributes(_attrLoc.key, entity.attrBufferData[_attrName].buffer,
entity.attrBufferData[_attrName].compCount);
}
}
//Bind triangle index buffer
gl.bindBuffer(GL.ELEMENT_ARRAY_BUFFER, entity.triBuffer);
//If already in use, is instanced material so skip configuration.
if(_materialInUse == entity.material){return;}
_materialInUse = entity.material;
//Use the material by applying it's specific uniforms
//Apply base color
gl.uniform4fv(programData.uniformData.uColor, entity.material.color);
//If shader uses a difuse texture
if(programData.uniformData.uDiffuseSampler){
//Store reference to make stack overflow lines shorter :-)
textureData = entity.material.diffuseTexture;
gl.activeTexture(gl.TEXTURE0);
//Use assigned texture
gl.bindTexture(gl.TEXTURE_2D, textureData);
//If this is a video, update the texture buffer using the current video's playback frame data
if(textureData.type == TEXTURE_TYPE.VIDEO &&
textureData.isLoaded &&
!textureData.paused){
//This accounts for 42% of all script execution time!!!
gl.texSubImage2D(gl.TEXTURE_2D, textureData.level, 0, 0,
textureData.width, textureData.height, textureData.internalFormat,
textureData.srcType, textureData.video);
}
gl.uniform1i(programData.uniformData.uDiffuseSampler, 0);
}
}
function _BindShaderAttributes(attrKey, buffer, compCount, type=GL.FLOAT, normalize=false, stride=0, offset=0){
gl.bindBuffer(GL.ARRAY_BUFFER, buffer);
gl.vertexAttribPointer(attrKey, compCount, type, normalize, stride, offset);
gl.enableVertexAttribArray(attrKey);
}
I've contemplated using pre-defined counters for all for loops to avoid the var i=0; allocation, but the gain from that seems hardly worth the effort.
Side Note, The source video is actually larger than 4K, but anything above 4K and FPS grinds to about 10-12.
Obligatory: The key functionality above is extracted from a larger WebGL rendering framework I wrote that itself runs pretty damn fast already. The reason I'm not 'just using' Three, AFrame, or other such common libraries is that they do not have an ATO from the DOD, whereas in-house developed code is ok.
Update 9/9/21: At some point when chrome updated from 90 to 93 the WebGL performance of texSubImage2D dropped dramatically, resulting in +100ms per frame execution regardless of CPU/GPU capability. Changing to use texImage2D now results in around 16ms per frame. In addition shifting from RGB to RGB565 offers up a few ms of performance while minimally sacrificing color.
I'd still love to hear from GL/WebGL experts as to what else I can do to improve performance.

Suggestion on how to create a window over a wall

I am working on a three.js application where I have to create a building structure (all on ground floor), the height, width, length will be specified by user. User can change wall and roof color (which are applied using texture, as I have images for each color with some texture). They can also add any accessory on a selected wall (like a window or a door), which can be then dragged and dropped on that same selected wall. After deciding where they want to put the window (for eg.) they will click a button to confirm the position. Now I have to create a window in the wall, so that I can see inside of the room. Please suggest your views on following approaches:
Once the user confirms the position of the door -
a.) I can add the mesh of the window in the main building mesh mainMesh.add(windowMesh);. But the problem is even if I set the transparent material to the window , the wall material still shows.
b.) I can subtract the window mesh from the main building mesh (using CSG, threeCSG) buildingmeshcsg.subtract(windowmeshcsg) which creates a hole in the building mesh, and then I put the window mesh over that hole. Now the problem is after any CSG operation the faces of the original geometry gets all mixed up, so after the csg operation, the color, UV of faces goes away.
c.) I can create wall in small sections,
like from one corner to window corners then, from another window corner to another wall corner. But this messes up the texture I have applied on walls, because I have created UV for front and back walls, as the texture was not applying correctly.
Please suggest your views.
Have to make something like this :https://forum.unity.com/threads/make-a-seethrough-window-without-making-hole-in-the-wall.286393/
THREE * (any version)
This sounds like a good candidate for the stencil buffer. Draw the window and write to stencil, draw the wall around the written values (0) then draw the wall within the written value (1).
These are the methods you are interested in:
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/stencilOp
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/stencilFunc
You need to do this first:
const gl = renderer.getContext() //obtain the actual webgl context, because three has no stencil functionality
And then you need to manage your render logic, this is where Object3D.onBeforeRender callback should help.
So let's assume that you have something like this:
const myWindow, myWall //two meshes, you loaded them, instantiated them
//for a proof of concept you can do something like this
const maskScene = new THREE.Scene()
const wallScene = new THREE.Scene()
const windowScene = new THREE.Scene()
maskScene.add(myWindow)
wallScene.add(myWall)
windowScene.add(myWindow)
render(){
gl.enable(gl.STENCIL_TEST) //enable stencil testing
gl.clearStencil( 0 ) //set stencil clear value
gl.clear( _gl.STENCIL_BUFFER_BIT ) //clear the stencil buffer with set value
gl.stencilFunc( gl.ALWAYS, 1, 1) //always pass the stencil test, with ref 1
gl.stencilOp( gl.REPLACE , gl.REPLACE , gl.REPLACE ) //replace the stencil value with the ref
gl.colorMask(false, false, false, false) //do not write any color
gl.depthMask(false) //do not write to depth
myRenderer.render(maskScene, myCamera) //only the stencil is drawn
//now you have a region in the frame buffer with stencil value of 1, and the rest 0, you can draw the wall in 0 and the window back at 1
gl.colorMask(true, true, true, true) //enable writing
gl.depthMask(true)
gl.stencilFunc( gl.EQUAL , 0 , 1 ) //set the stencil function to EQUAL
gl.stencilOp( gl.KEEP, gl.KEEP, gl.KEEP ) //keep the stencil value in all three tests
myRenderer.render( wallScene, myCamera ) //draw the wall where stencil is 0 (around the window)
gl.stencilFunc( gl.EQUAL , 1 , 1 ) // now func is EQUAL but to value 1
myRenderer.render( windowScene, myCamera ) //draw the window
}
This is the most basic attempt and it is not tested. Since it's working directly with the WebGL API it should work with any version of three. stencilOp takes two more arguments with which you can manage what happens with the depth tests.

Reading Pixels in WebGL 2 as Float values

I need to read the pixels of my framebuffer as float values.
My goal is to get a fast transfer of lots of particles between CPU and GPU and process them in realtime. For that I store the particle properties in a floating point texture.
Whenever a new particle is added, I want to get the current particle array back from the texture, add the new particle properties and then fit it back into the texture (this is the only way I could think of to dynamically add particles and process them GPU-wise).
I am using WebGL 2 since it supports reading back pixels to a PIXEL_PACK_BUFFER target. I test my code in Firefox Nightly. The code in question looks like this:
// Initialize the WebGLBuffer
this.m_particlePosBuffer = gl.createBuffer();
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, this.m_particlePosBuffer);
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);
...
// In the renderloop, bind the buffer and read back the pixels
gl.bindBuffer(gl.PIXEL_PACK_BUFFER, this.m_particlePosBuffer);
gl.readBuffer(gl.COLOR_ATTACHMENT0); // Framebuffer texture is bound to this attachment
gl.readPixels(0, 0, _texSize, _texSize, gl.RGBA, gl.FLOAT, 0);
I get this error in my console:
TypeError: Argument 7 of WebGLRenderingContext.readPixels could not be converted to any of: ArrayBufferView, SharedArrayBufferView.
But looking at the current WebGL 2 Specification, this function call should be possible. Using the type gl.UNSIGNED_BYTE also returns this error.
When I try to read the pixels in an ArrayBufferView (which I want to avoid since it seems to be way slower) it works with the format/type combination of gl.RGBA and gl.UNSIGNED_BYTE for a Uint8Array() but not with gl.RGBA and gl.FLOAT for a Float32Array() - this is as expected since it's documented in the WebGL Specification.
I am thankful for any suggestions on how to get my float pixel values from my framebuffer or on how to otherwise get this particle pipeline going.
Did you try using this extension?
var ext = gl.getExtension('EXT_color_buffer_float');
The gl you have is webgl1,not webgl2.Try:
var gl = document.getElementById("canvas").getContext('webgl2');
In WebGL2 the syntax for glReadPixel is
void gl.readPixels(x, y, width, height, format, type, ArrayBufferView pixels, GLuint dstOffset);
so
let data = new Uint8Array(gl.drawingBufferWidth * gl.drawingBufferHeight * 4);
gl.readPixels(0, 0, gl.drawingBufferWidth, gl.drawingBufferHeight, gl.RGBA, gl.UNSIGNED_BYTE, pixels, 0);
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/readPixels

android OpenGLES 1.x CameraPreview to Surfacetexture

I am trying to send the camera preview to a surfacetexture object and render it on a square. I have running code for GLES20 but didnt find anything for 1.x.
Basically it should work like this, right?
// setup texture
gl.glActiveTexture(GL10.GL_TEXTURE0);
gl.glGenTextures(1, textures, 0);
gl.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[0]);
gl.glTexParameterf(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, ...);
...
// setup surfacetexture object
surface = new SurfaceTexture(textures[0]);
surface.setOnFrameAvailableListener(this);
// setup camera
mCamera = Camera.open(0);
Camera.Parameters param = mCamera.getParameters();
List<Size> psize = param.getSupportedPreviewSizes();
//find previewsize to match glsurface from renderer
param.setPreviewSize(psize.get(i).width, psize.get(i).height);
mCamera.setParameters(param);
// set the texture and start preview
mCamera.setPreviewTexture(surface);
mCamera.startPreview();
// in the "onFrameAvailable" handler, i switch a flag to mark a new frame
updateSurface = true;
// and in the renderloop i update and redraw
if (updateSurface) {
surface.updateTexImage();
updateSurface = false;
}
gl.glActiveTexture(GL10.GL_TEXTURE0);
gl.glBindTexture(GLES11Ext.GL_TEXTURE_EXTERNAL_OES, textures[0]);
// Draw square
gl.glVertexPointer(3, GL10.GL_FLOAT, 0, vertexBufferFloor);
gl.glTexCoordPointer(2, GL10.GL_FLOAT, 0, textureBuffer);
gl.glDrawArrays(GL10.GL_TRIANGLE_STRIP, 0, vertices.length / 3);
The square gets drawn but is completely white. I dont receive glErrors or other exceptions. The "onFrameAvailable" handler gets called too.
If i use glTeximage with a loaded bitmap, it is correctly drawn on the square.
ANY ideas? Thank you!
I'm facing the same problem. Maybe I'm wrong, but it seems that SurfaceTexture is not compatible with GLES10. Surface texture uses GL_TEXTURE_EXTERNAL_OES, thereby it a custom fragment shader that is able to use this texture ("#extension GL_OES_EGL_image_external : require ").
As glUseProgram(...), etc are not avaible in GLES10, we cannot use custom shaders.
As I said, maybe I'm wrong... Good luck
EDIT : I finaly get it to work. You should use "gl.glEnable(GLES11Ext.GL_TEXTURE_EXTERNAL_OES);"

How to use Transformation Matrix for SpriteBatch correctly?

I'm new to XNA and would like to develop a light-weight 2D engine over it, with the entities organized into parent-child hierarchy. I think of matrix when drawing children, because their position, rotation and scale are depend on their parent.
If I use SpriteBatch.Begin(), my rectangles can be drawn on the screen, but when I change them into:
this.DrawingMatrix = Matrix.Identity;
this.SpriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.LinearClamp, DepthStencilState.None, RasterizerState.CullClockwise, null, this.DrawingMatrix);
nothing is drawn anymore. I even tried new Matrix() or Matrix.CreateTranslation(0, 0, 0) for DrawingMatrix.
My first question is: why doesn't it work? I'm not working with any camera or viewport.
Secondly, before drawing an entity, I call the PreDraw to transform the matrix (I will then reset to original state at PostDraw):
protected virtual void PreDraw(Engine pEngine)
{
pEngine.DrawingMatrix *=
Matrix.CreateTranslation(this.X, this.Y, 0) *
Matrix.CreateScale(this.ScaleX, this.ScaleY, 1) *
Matrix.CreateRotationZ(this.Rotation);
}
Please clarify the correction of above code. And I need to scale not at the origin, but at ScaleCenterX and ScaleCenterY, how can I achieve this?
ADDED: Here is an example of my engine's draw process:
Call these code:
this.DrawingMatrix = Matrix.CreateTranslation(0, 0, 0);
this.SpriteBatch.Begin(SpriteSortMode.Deferred, BlendState.AlphaBlend, SamplerState.LinearClamp, DepthStencilState.None, RasterizerState.CullClockwise, null, this.DrawingMatrix);
Call PreDraw(), with is:
protected virtual void PreDraw(Engine pEngine)
{
pEngine.DrawingMatrix *=
Matrix.CreateTranslation(this.X, this.Y, 0) *
Matrix.CreateScale(this.ScaleX, this.ScaleY, 1) *
Matrix.CreateRotationZ(this.Rotation);
}
Call Draw(), for example, in my Rect class:
protected override void Draw(Engine pEngine)
{
pEngine.SpriteBatch.Draw(pEngine.RectangleTexture, new Rectangle(0, 0, (int)this.Width, (int)this.Height), new Rectangle(0, 0, 1, 1), this.Color);
}
If I replace above Begin code with this.SpriteBatch.Begin(), the rectangle is drawn correctly, so I guess it is because of the matrix.
First issue is a simple bug: The default for SpriteBatch is CullCounterClockwise, but you have specified CullClockwise causing all your sprites to get back-face-culled. You can pass null if you just want to use the default render states - you don't need to specify them explicitly.
(You would need to change the cull mode if you used a negative scale.)
To answer your second question: You need to translate "back" to place the scaling origin (your ScaleCenterX and ScaleCenterY) at the world origin (0,0). Transformations always happen around (0,0). So normally the order is: translate sprite origin back to the world origin, scale, rotate, translate to place sprite origin at desired world position.
Also, I hope that your PostDraw is not applying the reverse transformations (you made it sound like it does). That is very likely to cause precision problems. You should save and restore the matrix instead.

Resources