I am using glTF-transform to create GLB (compressed by gltf-pack), and encountered strange behavior in THREE.GLTFLoader.
I am using vertex colors and passing them this way
My pipeline looks like:
Create Glb -> gltfpack -> load to THREE using gltfLoader -> mergeBufferGeometry -> render() with opacity in fragmentShader
const primitive = doc
.createPrimitive()
.setAttribute('POSITION', position)
.setIndices(indices);
const positionArr = position.getArray();
if (positionArr) {
const colors = [];
const colorAccessor = doc
.createAccessor()
.setType('VEC4')
.setBuffer(buffer);
const rgba = this.toRGBAColor(Number.parseInt(sPrim.c));
for (let i = 0; i < position.getCount(); i++) {
colors.push(rgba[0], rgba[1], rgba[2], rgba[3]);
}
colorAccessor.setArray(new Float32Array(colors));
primitive.setAttribute('COLOR', colorAccessor);
}
mesh.addPrimitive(primitive);
In this case, the colors are correct, RGBA from 0 to 1
console.log(primitive.getAttribute('COLOR')?.getArray());
-> Float32Array(16) [
0.4627451002597809, 0.27450981736183167,
0.20000000298023224, 1,
0.4627451002597809, 0.27450981736183167,
0.20000000298023224, 1,
0.4627451002597809, 0.27450981736183167,
0.20000000298023224, 1,
0.4627451002597809, 0.27450981736183167,
0.20000000298023224, 1
]
But, when I load the model into gltfLoader, the color attribute array becomes Uint8Array (with values 0-255 even for alpha)
So, what is the correct way to pass the color of the vertices?
COLOR_0 (not COLOR) is the expected name of the vertex color attribute – the model could have more than one vertex color set. gltfpack is "normalizing" the float32 colors, which means it replaces float32 values in [0, 1] with uint8 values in [0, 255] instead, requiring ~75% less space. three.js can support both.
Related
I need to find the objects that fully/partly visible on the rendered screen. I know this can be done by coloring each object uniquely, rendering the scene, and detecting the colors that end up on the screen. This is a screen-space operation that would involve fiddling with the frame-buffer. Are there any special functions/helpers within three.js that do this more easily?
You can check if object is in view frustum of the camera. See Frustum in Three.js documentation.
One way to achieve this is to render your scene once with constant shading, colour-coding your objects as you need, with any anti-aliasing and other effects turned off, so that you can easily map a read pixel back to its object by its colour.
Then, you can read pixels from your render target, for which you can use three.js' WebGLRenderer.readRenderTargetPixels() (see docs). You can then read the colours out of the buffer you pass to it.
Something like this:
// Render your scene first, into a renderTarget. Then:
const buffer = new Uint8Array(width * height * 4);
this.renderer.readRenderTargetPixels(renderTarget, 0, 0, width, height, buffer);
for (let i=0; i<buffer.length/4; ++i) {
const r = buffer[i*4 ];
const g = buffer[i*4 + 1];
const b = buffer[i*4 + 2];
const rgb = (r << 16) | (g << 8) | b;
// Do your mapping
}
This is very much just WebGL though, and don't know whether there might be a better way to do this within three.js.
I have a total of two textures, the first is used as a framebuffer to work with inside a computeshader, which is later blitted using BlitFramebuffer(...). The second is supposed to be an OpenGL array texture, which is used to look up textures and copy them onto the framebuffer. It's created in the following way:
var texarray uint32
gl.GenTextures(1, &texarray)
gl.ActiveTexture(gl.TEXTURE0 + 1)
gl.BindTexture(gl.TEXTURE_2D_ARRAY, texarray)
gl.TexParameteri(gl.TEXTURE_2D_ARRAY, gl.TEXTURE_MIN_FILTER, gl.LINEAR)
gl.TexImage3D(
gl.TEXTURE_2D_ARRAY,
0,
gl.RGBA8,
16,
16,
22*48,
0,
gl.RGBA, gl.UNSIGNED_BYTE,
gl.Ptr(sheet.Pix))
gl.BindImageTexture(1, texarray, 0, false, 0, gl.READ_ONLY, gl.RGBA8)
sheet.Pix is just the pixel array of an image loaded as a *image.NRGBA
The compute-shader looks like this:
#version 430
layout(local_size_x = 1, local_size_y = 1) in;
layout(rgba32f, binding = 0) uniform image2D img;
layout(binding = 1) uniform sampler2DArray texAtlas;
void main() {
ivec2 iCoords = ivec2(gl_GlobalInvocationID.xy);
vec4 c = texture(texAtlas, vec3(iCoords.x%16, iCoords.y%16, 7));
imageStore(img, iCoords, c);
}
When i run the program however, the result is just a window filled with the same color:
So my question is: What did I do wrong during the shader creation and what needs to be corrected?
For any open code questions, here's the corresponding repo
vec4 c = texture(texAtlas, vec3(iCoords.x%16, iCoords.y%16, 7))
That can't work. texture samples the texture at normalized coordinates, so the texture is in [0,1] (in the st domain, the third dimension is the layer and is correct here), coordinates outside of that ar handled via the GL_WRAP_... modes you specified (repeat, clamp to edge, clamp to border). Since int % 16 is always an integer, and even with repetition only the fractional part of the coordinate will matter, you are basically sampling the same texel over and over again.
If you need the full texture sampling (texture filtering, sRGB conversions etc.), you have to use the normalized coordinates instead. But if you only want to access individual texel data, you can use texelFetch and keep the integer data instead.
Note, since you set the texture filter to GL_LINEAR, you seem to want filtering, however, your coordinates appear as if you would want at to access the texel centers, so if you're going the texture route , thenvec3(vec2(iCoords.xy)/vec2(16) + vec2(1.0/32.0) , layer) would be the proper normalization to reach the texel centers (together with GL_REPEAT), but then, the GL_LINEAR filtering would yield identical results to GL_NEAREST.
I am trying to construct a right-view image from a left-view image and its disparity map. I use the middleburry dataset 2003 (http://vision.middlebury.edu/stereo/data/scenes2003/) with the full size images, which means the value v of each pixel in the disparity map corresponds to a shift of v pixels on the left-view image.
My algorithm is quite simple. For each pixel of coordinates (x, y) in the left-view image, I copy this pixel on the right-view image but at the coordinates (x - d, y) where d is the value of the disparity map at the coordinates (x, y). If the disparity value is 0, I just don't do anything. I use openCV to manipulate the images.
Here is my code:
void computeCorrespondingImage(const cv::Mat &img, const cv::Mat &disparity, cv::Mat &dest,
const bool leftInput, const int disparityScale)
{
const int shiftDirection = leftInput ? -1 : 1;
dest.create(img.rows, img.cols, img.type());
for (int i(0) ; i < img.rows ; ++i) {
for (int j(0) ; j < img.cols ; ++j) {
const uchar d(disparity.at<const uchar>(i, j));
const int computedColumn(j + shiftDirection * (d / disparityScale));
// No need to consider pixels who would be outside of the image's bounds
if (d > 0 && computedColumn >= 0 && computedColumn < img.cols) {
dest.at<cv::Vec3b>(i, computedColumn) = img.at<const cv::Vec3b>(i, j);
}
}
}
}
Since the disparity map is a ground-truth disparity map, I would expect to get an image quite like the right-view image provided in the dataset with some black areas (for which the disparity is unknown).
However, for some reasons it's like the computed right-view image is split at the center, making the image unusable.
Left-view image :
Ground-truth disparity map :
What I get :
Thank you in advance for your help.
Ok, I figured it out. I was loading the disparity image with imread without specifying that it was a gray scale image (with IMREAD_GRAYSCALE). Therefore, openCV loaded it as an RGB image and when I was accessing a pixel of the disparity with at(), I was specifying uchar as the wanted type. So I guess there was kind of a conversion from Vec3b to uchar that gave false values.
I display a "curved tube" and color its vertices based on their distance to the plane the curve lays on.
It works mostly fine, however, when I reduce the resolution of the tube, artifacts starts to appear in the tube colors.
Those artifacts seem to depend on the camera position. If I move the camera around, sometimes the artifacts disappear. Not sure it makes sense.
Live demo: http://jsfiddle.net/gz1wu369/15/
I do not know if there is actually a problem in the interpolation or if it is just a "screen" artifact.
Afterwards I render the scene to a texture, looking at it from the "top". It then looks like a "deformation" field that I use in another shader, hence the need for continuous color.
I do not know if it is the expected behavior or if there is a problem in my code while setting the vertices color.
Would using the THREEJS Extrusion tools instead of the tube geometry solve my issue?
const tubeGeo = new THREE.TubeBufferGeometry(closedSpline, steps, radius, curveSegments, false);
const count = tubeGeo.attributes.position.count;
tubeGeo.addAttribute('color', new THREE.BufferAttribute(new Float32Array(count * 3), 3));
const colors = tubeGeo.attributes.color;
const color = new THREE.Color();
for (let i = 0; i < count; i++) {
const pp = new THREE.Vector3(
tubeGeo.attributes.position.array[3 * i],
tubeGeo.attributes.position.array[3 * i + 1],
tubeGeo.attributes.position.array[3 * i + 2]);
const distance = plane.distanceToPoint(pp);
const normalizedDist = Math.abs(distance) / radius;
const t2 = Math.floor(i / (curveSegments + 1));
color.setHSL(0.5 * t2 / steps, .8, .5);
const green = 1 - Math.cos(Math.asin(Math.abs(normslizedDist)));
colors.setXYZ(i, color.r, green, 0);
}
Low-res tubes with "Normals" material shows different artifact
High resolution tube hide the artifacts:
I have a sample image and a target image. I want to transfer the color shades of sample image to target image. Please tell me how to extract the color from sample image.
Here the images:
input source image:
input map for desired output image
output image
You can use a technique called "Histogram matching" (another description)
Basically, you use the histogram for your source image as a goal and transform the values for each input map pixel to get the output histogram as close to source as possible. You do it for each rgb channel of the image.
Here is my python code for that:
from scipy.misc import imsave, imread
import numpy as np
imsrc = imread("source.jpg")
imtint = imread("tint_target.jpg")
nbr_bins=255
imres = imsrc.copy()
for d in range(3):
imhist,bins = np.histogram(imsrc[:,:,d].flatten(),nbr_bins,normed=True)
tinthist,bins = np.histogram(imtint[:,:,d].flatten(),nbr_bins,normed=True)
cdfsrc = imhist.cumsum() #cumulative distribution function
cdfsrc = (255 * cdfsrc / cdfsrc[-1]).astype(np.uint8) #normalize
cdftint = tinthist.cumsum() #cumulative distribution function
cdftint = (255 * cdftint / cdftint[-1]).astype(np.uint8) #normalize
im2 = np.interp(imsrc[:,:,d].flatten(),bins[:-1],cdfsrc)
im3 = np.interp(imsrc[:,:,d].flatten(),cdftint, bins[:-1])
imres[:,:,d] = im3.reshape((imsrc.shape[0],imsrc.shape[1] ))
imsave("histnormresult.jpg", imres)
The output for you samples will look like that:
You could also try making the same in HSV colorspace - it might give better results.
I think the hardest part is to determine the dominant color of the first image. Just looking at it, with all the highlights and shadows, the best overall color will be the one that has the highest combination of brightness and saturation. I start with a blurred image to reduce the effects of noise and other anomalies, then convert each pixel to the HSV color space for the brightness and saturation measurement. Here's how it looks in Python with PIL and colorsys:
blurred = im1.filter(ImageFilter.BLUR)
ld = blurred.load()
max_hsv = (0, 0, 0)
for y in range(blurred.size[1]):
for x in range(blurred.size[0]):
r, g, b = tuple(c / 255. for c in ld[x, y])
h, s, v = colorsys.rgb_to_hsv(r, g, b)
if s + v > max_hsv[1] + max_hsv[2]:
max_hsv = h, s, v
r, g, b = tuple(int(c * 255) for c in colorsys.hsv_to_rgb(*max_hsv))
For your image I get a color of (210, 61, 74) which looks like:
From that point it's just a matter of transferring the hue and saturation to the other image.
The histogram matching solutions above did not work for me. Here is my own, based on OpenCV:
def match_image_histograms(image, reference):
chans1 = cv2.split(image)
chans2 = cv2.split(reference)
new_chans = []
for ch1, ch2 in zip(chans1, chans2):
hist1 = cv2.calcHist([ch1], [0], None, [256], [0, 256])
hist1 /= hist1.sum()
hist2 = cv2.calcHist([ch2], [0], None, [256], [0, 256])
hist2 /= hist2.sum()
lut = np.searchsorted(hist1.cumsum(), hist2.cumsum())
new_chans.append(cv2.LUT(ch1, lut))
return cv2.merge(new_chans).astype('uint8')
obtain average color from color map
ignore saturated white/black colors
convert light map to grayscale
change dynamic range of lightmap to match your desired output
I use max dynamic range. You could compute the range of color map and set it for light map
multiply the light map by avg color
This is how it looks like:
And this is the C++ source code
//picture pic0,pic1,pic2;
// pic0 - source color
// pic1 - source light map
// pic2 - output
int x,y,rr,gg,bb,i,i0,i1;
double r,g,b,a;
// init output as source light map in grayscale i=r+g+b
pic2=pic1;
pic2.rgb2i();
// change light map dynamic range to maximum
i0=pic2.p[0][0].dd; // min
i1=pic2.p[0][0].dd; // max
for (y=0;y<pic2.ys;y++)
for (x=0;x<pic2.xs;x++)
{
i=pic2.p[y][x].dd;
if (i0>i) i0=i;
if (i1<i) i1=i;
}
for (y=0;y<pic2.ys;y++)
for (x=0;x<pic2.xs;x++)
{
i=pic2.p[y][x].dd;
i=(i-i0)*767/(i1-i0);
pic2.p[y][x].dd=i;
}
// extract average color from color map (normalized to unit vecotr)
for (r=0.0,g=0.0,b=0.0,y=0;y<pic0.ys;y++)
for (x=0;x<pic0.xs;x++)
{
rr=BYTE(pic0.p[y][x].db[picture::_r]);
gg=BYTE(pic0.p[y][x].db[picture::_g]);
bb=BYTE(pic0.p[y][x].db[picture::_b]);
i=rr+gg+bb;
if (i<400) // ignore saturated colors (whiteish) 3*255=white
if (i>16) // ignore too dark colors (whiteish) 0=black
{
r+=rr;
g+=gg;
b+=bb;
}
}
a=1.0/sqrt((r*r)+(g*g)+(b*b)); r*=a; g*=a; b*=a;
// recolor output
for (y=0;y<pic2.ys;y++)
for (x=0;x<pic2.xs;x++)
{
a=DWORD(pic2.p[y][x].dd);
rr=r*a; if (rr>255) rr=255; pic2.p[y][x].db[picture::_r]=BYTE(rr);
gg=g*a; if (gg>255) gg=255; pic2.p[y][x].db[picture::_g]=BYTE(gg);
bb=b*a; if (bb>255) bb=255; pic2.p[y][x].db[picture::_b]=BYTE(bb);
}
I am using own picture class so here some members:
xs,ys size of image in pixels
p[y][x].dd is pixel at (x,y) position as 32 bit integer type
p[y][x].db[4] is pixel access by color bands (r,g,b,a)
[notes]
If this does not meet your needs then please specify more and add more images. Because your current example is really not self explanatonary
Regarding previous answer, one thing to be careful with:
once the CDF will reach its maximum (=1), the interpolation will get mislead and will match wrongly your values. To avoid this, you should provide the interpolation function only the part of CDF meaningful (not after where it reaches 1) and the corresponding bins. Here the answer adapted:
from scipy.misc import imsave, imread
import numpy as np
imsrc = imread("source.jpg")
imtint = imread("tint_target.jpg")
nbr_bins=255
imres = imsrc.copy()
for d in range(3):
imhist,bins = np.histogram(imsrc[:,:,d].flatten(),nbr_bins,normed=True)
tinthist,bins = np.histogram(imtint[:,:,d].flatten(),nbr_bins,normed=True)
cdfsrc = imhist.cumsum() #cumulative distribution function
cdfsrc = (255 * cdfsrc / cdfsrc[-1]).astype(np.uint8) #normalize
cdftint = tinthist.cumsum() #cumulative distribution function
cdftint = (255 * cdftint / cdftint[-1]).astype(np.uint8) #normalize
im2 = np.interp(imsrc[:,:,d].flatten(),bins[:-1],cdfsrc)
if (cdftint==1).sum()>0:
idx_max = np.where(cdftint==1)[0][0]
im3 = np.interp(im2,cdftint[:idx_max+1], bins[:idx_max+1])
else:
im3 = np.interp(im2,cdftint, bins[:-1])
Enjoy!