I used Blender to generate some color images and their corresponding depth map, along with their camera properties(intrinsic and extrinsic).
Then I want to use these information to generate a 3D point cloud from these 2D images using 2D to 3D projection techniques.
Here is the viewpoint of one the cameras in Blender.
I wanted to have the rotation and translation matrix of the camera.
I used the code in this link camera matrix for Blender written by #rfabbri and I used this method "get_3x4_RT_matrix_from_blender" to have the rotation matrix.
After that I want to do 2D to 3D projection with all of these information.
For 2D to 3D projection, I wrote the following code in Java:
static double[] projUVZtoXY( double u, double v, double d)
{
// "u" and "v" are the pixel number of 2D image and
// "d" is the depth of this pixel (distance of the point to camera)
double[] p = new double[]{u, v, 1};
double[] translate = calibStruct.getM_Trans(); // Translation Matrix, from **T_world2cv** matrix in **get_3x4_RT_matrix_from_blender** method
double[] rotation = calibStruct.getM_RotMatrix(); // Rotation Matrix, from **R_world2cv** matrix in **get_3x4_RT_matrix_from_blender** method
double[] K = calibStruct.getM_K(); // Intrinsic Matrix, from K matrix in **get_calibration_matrix_K_from_blender** method
double[][] invertR = invert33(rotation); // This method give me R^-1 matrix
double[][] invertK = invert33(K); // This method give me K^-1 matrix
double[][] invK_mul_depth = multiply33_scalar(invertK, d); // this method multiply scalar value of "d" to "invertK" matrix
double[] invK_mul_depth_p = multiply33_31(invK_mul_depth, p); // this method multiply 3*3 matrix of "invK_mul_depth" by 3*1 matrix of "p"
// subtract translation Matrix from the "invK_mul_depth_p" matrix
double[] d_InvK_p_trans = new double[]{invK_mul_depth_p[0] - translate[0],
invK_mul_depth_p[1] - translate[1],
invK_mul_depth_p[2] - translate[2]};
double[] xyz = multiply33_31(invertR, d_InvK_p_trans );
return xyz;
}
All the above code is trying to implement this 3D warping algorithm, to project uv pixel to XYZ 3D point.
But when I generate the 3D point cloud, it looks like this: (in Meshlab)
It is the point cloud of just one image, the following image:
I can't understand what happened here. And why in 3D point Cloud, all of the players in the image, are repeated in a line !
Could anyone guess what is happening?
I think maybe the Rotation Matrix that I get it from Blender, is not correct. What's your idea?
Thanks,
Mozhde
Related
Assume I have a 2x2 matrix filled with values which will represent a plane. Now I want to rotate the plane around itself in a 3-D way, in the "z-Direction". For a better understanding, see the following image:
I wondered if this is possible by a simple affine matrix, thus I created the following simple script:
%Create a random value matrix
A = rand*ones(200,200);
%Make a box in the image
A(50:200-50,50:200-50) = 1;
Now I can apply transformations in the 2-D room simply by a rotation matrix like this:
R = affine2d([1 0 0; .5 1 0; 0 0 1])
tform = affine3d(R);
transformed = imwarp(A,tform);
However, this will not produce the desired output above, and I am not quite sure how to create the 2-D affine matrix to create such behavior.
I guess that a 3-D affine matrix can do the trick. However, if I define a 3-D affine matrix I cannot work with the 2-D representation of the matrix anymore, since MATLAB will throw the error:
The number of dimensions of the input image A must be 3 when the
specified geometric transformation is 3-D.
So how can I code the desired output with an affine matrix?
The answer from m3tho correctly addresses how you would apply the transformation you want: using fitgeotrans with a 'projective' transform, thus requiring that you specify 4 control points (i.e. 4 pairs of corresponding points in the input and output image). You can then apply this transform using imwarp.
The issue, then, is how you select these pairs of points to create your desired transformation, which in this case is to create a perspective projection. As shown below, a perspective projection takes into account that a viewing position (i.e. "camera") will have a given view angle defining a conic field of view. The scene is rendered by taking all 3-D points within this cone and projecting them onto the viewing plane, which is the plane located at the camera target which is perpendicular to the line joining the camera and its target.
Let's first assume that your image is lying in the viewing plane and that the corners are described by a normalized reference frame such that they span [-1 1] in each direction. We need to first select the degree of perspective we want by choosing a view angle and then computing the distance between the camera and the viewing plane. A view angle of around 45 degrees can mimic the sense of perspective of normal human sight, so using the corners of the viewing plane to define the edge of the conic field of view, we can compute the camera distance as follows:
camDist = sqrt(2)./tand(viewAngle./2);
Now we can use this to generate a set of control points for the transformation. We first apply a 3-D rotation to the corner points of the viewing plane, rotating around the y axis by an amount theta. This rotates them out of plane, so we now project the corner points back onto the viewing plane by defining a line from the camera through each rotated corner point and finding the point where it intersects the plane. I'm going to spare you the mathematical derivations (you can implement them yourself from the formulas in the above links), but in this case everything simplifies down to the following set of calculations:
term1 = camDist.*cosd(theta);
term2 = camDist-sind(theta);
term3 = camDist+sind(theta);
outP = [-term1./term2 camDist./term2; ...
term1./term3 camDist./term3; ...
term1./term3 -camDist./term3; ...
-term1./term2 -camDist./term2];
And outP now contains your normalized set of control points in the output image. Given an image of size s, we can create a set of input and output control points as follows:
scaledInP = [1 s(1); s(2) s(1); s(2) 1; 1 1];
scaledOutP = bsxfun(#times, outP+1, s([2 1])-1)./2+1;
And you can apply the transformation like so:
tform = fitgeotrans(scaledInP, scaledOutP, 'projective');
outputView = imref2d(s);
newImage = imwarp(oldImage, tform, 'OutputView', outputView);
The only issue you may come across is that a rotation of 90 degrees (i.e. looking end-on at the image plane) would create a set of collinear points that would cause fitgeotrans to error out. In such a case, you would technically just want a blank image, because you can't see a 2-D object when looking at it edge-on.
Here's some code illustrating the above transformations by animating a spinning image:
img = imread('peppers.png');
s = size(img);
outputView = imref2d(s);
scaledInP = [1 s(1); s(2) s(1); s(2) 1; 1 1];
viewAngle = 45;
camDist = sqrt(2)./tand(viewAngle./2);
for theta = linspace(0, 360, 360)
term1 = camDist.*cosd(theta);
term2 = camDist-sind(theta);
term3 = camDist+sind(theta);
outP = [-term1./term2 camDist./term2; ...
term1./term3 camDist./term3; ...
term1./term3 -camDist./term3; ...
-term1./term2 -camDist./term2];
scaledOutP = bsxfun(#times, outP+1, s([2 1])-1)./2+1;
tform = fitgeotrans(scaledInP, scaledOutP, 'projective');
spinImage = imwarp(img, tform, 'OutputView', outputView);
if (theta == 0)
hImage = image(spinImage);
set(gca, 'Visible', 'off');
else
set(hImage, 'CData', spinImage);
end
drawnow;
end
And here's the animation:
You can perform a projective transformation that can be estimated using the position of the corners in the first and second image.
originalP='peppers.png';
original = imread(originalP);
imshow(original);
s = size(original);
matchedPoints1 = [1 1;1 s(1);s(2) s(1);s(2) 1];
matchedPoints2 = [1 1;1 s(1);s(2) s(1)-100;s(2) 100];
transformType = 'projective';
tform = fitgeotrans(matchedPoints1,matchedPoints2,'projective');
outputView = imref2d(size(original));
Ir = imwarp(original,tform,'OutputView',outputView);
figure; imshow(Ir);
This is the result of the code above:
Original image:
Transformed image:
I have initiated a PIXI js canvas:
g_App = new PIXI.Application(800, 600, { backgroundColor: 0x1099bb });
Set up a container:
container = new PIXI.Container();
g_App.stage.addChild(container);
Put a background texture (2000x2000) into the container:
var texture = PIXI.Texture.fromImage('picBottom.png');
var back = new PIXI.Sprite(texture);
container.addChild(back);
Set the global:
var g_Container = container;
I do various pivot points and rotations on container and canvas stage element:
// Set the focus point of the container
g_App.stage.x = Math.floor(400);
g_App.stage.y = Math.floor(500); // Note this one is not central
g_Container.pivot.set(1000, 1000);
g_Container.rotation = 1.5; // radians
Now I need to be able to convert a canvas pixel to the pixel on the background texture.
g_Container has an element transform which in turn has several elements localTransform, pivot, position, scale ands skew. Similarly g_App.stage has the same transform element.
In Maths this is simple, you just have vector point and do matix operations on them. Then to go back the other way you just find inverses of those matrices and multiply backwards.
So what do I do here in pixi.js?
How do I convert a pixel on the canvas and see what pixel it is on the background container?
Note: The following is written using the USA convention of using matrices. They have row vectors on the left and multiply them by the matrix on the right. (Us pesky Brits in the UK do the opposite. We have column vectors on the right and multiply it by the matrix on the left. This means UK and USA matrices to do the same job will look slightly different.)
Now I have confused you all, on with the answer.
g_Container.transform.localTransform - this matrix takes the world coords to the scaled/transposed/rotated COORDS
g_App.stage.transform.localTransform - this matrix takes the rotated world coords and outputs screen (or more accurately) html canvas coords
So for example the Container matrix is:
MatContainer = [g_Container.transform.localTransform.a, g_Container.transform.localTransform.b, 0]
[g_Container.transform.localTransform.c, g_Container.transform.localTransform.d, 0]
[g_Container.transform.localTransform.tx, g_Container.transform.localTransform.ty, 1]
and the rotated container matrix to screen is:
MatToScreen = [g_App.stage.transform.localTransform.a, g_App.stage.transform.localTransform.b, 0]
[g_App.stage.transform.localTransform.c, g_App.stage.transform.localTransform.d, 0]
[g_App.stage.transform.localTransform.tx, g_App.stage.transform.localTransform.ty, 1]
So to get from World Coordinates to Screen Coordinates (noting our vector will be a row on the left, so the first operation matrix that acts first on the World coordinates must also be on the left), we would need to multiply the vector by:
MatAll = MatContainer * MatToScreen
So if you have a world coordinate vector vectWorld = [worldX, worldY, 1.0] (I'll explain the 1.0 at the end), then to get to the screen coords you would do the following:
vectScreen = vectWorld * MatAll
So to get screen coords and to get to world coords we first need to calculate the inverse matrix of MatAll, call it invMatAll. (There are loads of places that tell you how to do this, so I will not do it here.)
So if we have screen (canvas) coordinates screenX and screenY, we need to create a vector vectScreen = [screenX, screenY, 1.0] (again I will explain the 1.0 later), then to get to world coordinates worldX and worldY we do:
vectWorld = vectScreen * invMatAll
And that is it.
So what about the 1.0?
In a 2D system you can do rotations, scaling with 2x2 matrices. Unfortunately you cannot do a 2D translations with a 2x2 matrix. Consequently you need 3x3 matrices to fully describe all 2D scaling, rotations and translations. This means you need to make your vector 3D as well, and you need to put a 1.0 in the third position in order to do the translations properly. This 1.0 will also be 1.0 after any matrix operation as well.
Note: If we were working in a 3D system we would need 4x4 matrices and put a dummy 1.0 in our 4D vectors for exactly the same reasons.
I'm about to project image into cylindrical panorama. But first I need to get the pixel (or color from pixel) I'm going to draw, then then do some Math in shaders with polar coordinates to get new position of pixel and then finally draw pixel.
Using this way I'll be able to change shape of image from polygon shape to whatever I want.
But I cannot find anything about this method (get pixel first, then do the Math and get new position for pixel).
Is there something like this, please?
OpenGL historically doesn't work that way around; it forward renders — from geometry to pixels — rather than backwards — from pixel to geometry.
The most natural way to achieve what you want to do is to calculate texture coordinates based on geometry, then render as usual. For a cylindrical mapping:
establish a mapping from cylindrical coordinates to texture coordinates;
with your actual geometry, imagine it placed within the cylinder, then from each vertex proceed along the normal until you intersect the cylinder. Use that location to determine the texture coordinate for the original vertex.
The latter is most easily and conveniently done within your geometry shader; it's a simple ray intersection test, with attributes therefore being only vertex location and vertex normal, and texture location being a varying that is calculated purely from the location and normal.
Extemporaneously, something like:
// get intersection as if ray hits the circular region of the cylinder,
// i.e. where |(position + n*normal).xy| = 1
float planarLengthOfPosition = length(position.xy);
float planarLengthOfNormal = length(normal.xy);
float planarDistanceToPerimeter = 1.0 - planarLengthOfNormal;
vec3 circularIntersection = position +
(planarDistanceToPerimeter/planarLengthOfNormal)*normal;
// get intersection as if ray hits the bottom or top of the cylinder,
// i.e. where |(position + n*normal).z| = 1
float linearLengthOfPosition = abs(position.z);
float linearLengthOfNormal = abs(normal.z);
float linearDistanceToEdge = 1.0 - linearLengthOfPosition;
vec3 endIntersection = position +
(linearDistanceToEdge/linearLengthOfNormal)*normal;
// pick whichever of those was lesser
vec3 cylindricalIntersection = mix(circularIntersection,
endIntersection,
step(linearDistanceToEdge,
planarDistanceToPerimeter));
// ... do something to map cylindrical intersection to texture coordinates ...
textureCoordinateVarying =
coordinateFromCylindricalPosition(cylindricalIntersection);
With a common implementation of coordinateFromCylindricalPosition possibly being simply return vec2(atan(cylindricalIntersection.y, cylindricalIntersection.x) / 6.28318530717959, cylindricalIntersection.z * 0.5);.
i have three 3d (x,y,z) points.
i get them by tracking the corners an object with a kinect.
i now want to translate and rotate a 3d model accordingly.
i get roll and pitch by doing this:
(i am using openframeworks.cc so some of the class methods might seem strange to people)
ofVec3f v10 = pointB - pointA;
ofVec3f v20 = pointC - pointA;
v10.normalize();
v20.normalize();
//create rotation matrix for roll+pitch relative to up vector 0,0,1
ofVec3f normaleVec = v10.crossed(v20);
ofVec3f fromVec = ofVec3f(0,0,1);
ofVec3f toVec = normaleVec;
mMR0.makeRotationMatrix(fromVec,toVec);
to get the heading / yaw i do this:
ofVec3f myV0_flat = avePointA*mMR0.getInverse();
ofVec3f myV1_flat = avePointB*mMR0.getInverse();
//get points relative to origion
ofVec3f myV10_flat = myV1_flat - myV0_flat;
//create rotation matrix for heading relative to flat 2d plane
float angle = atan2(myV10_flat.x,myV10_flat.y)/M_PI*180;
mMR1.makeRotationMatrix(angle,fromVec);
and finally create translation matrix and combine all the matrices:
mMT1.makeTranslationMatrix(avePointD); //translate from origin
ofMatrix4x4 mMc;
mMc = mMR0 * mMR1 * mMT1;
but when my 3d model rotates around it seems to dip always at the same angle.
my question is. how would i calculate roll and pitch separately, so i can where it dips and how to fix it.
thx.
s.
I'm attempting to use Java3D to rotate a vector. My goal is create a transform that will make the vector parallel with the y-axis. To do this, I calculated the angle between the original vector and an identical vector except that it has a z value of 0 (original x, original y, 0 for z-value). I then did the same thing for the y-axis (original x, 0 for y-value, original z). I then used each angle to create two Transform3D objects, multiply them together and apply to the vector. My code is as follows:
Transform3D yRotation = new Transform3D();
Transform3D zRotation = new Transform3D();
//create new normal vector
Vector3f normPoint = new Vector3f (normal.getX(), normal.getY(), normal.getZ());
//****Z rotation methods*****
Vector3f newNormPointZ = new Vector3f(normal.getX(), normal.getY(),0.0F);
float zAngle = normPoint.angle(newNormPointZ);
zRotation.rotZ(zAngle);
//****Y rotation methods*****
Vector3f newNormPointY = new Vector3f(normal.getX(),0.0F, normal.getZ());
float yAngle = normPoint.angle(newNormPointY);
yRotation.rotY(yAngle);
//combine the two rotations
yRotation.mul(zRotation);
System.out.println("before trans normal = " +normPoint.x + ", "+normPoint.y+", "+normPoint.z);
//PRINT STATEMENT RETURNS: before trans normal = 0.069842085, 0.99316376, 0.09353002
//perform transform
yRotation.transform(normPoint);
System.out.println("normal trans = " +normPoint.x + ", "+normPoint.y+", "+normPoint.z);
//PRINT STATEMENT RETURNS: normal trans = 0.09016449, 0.99534255, 0.03411238
I was hoping the transform would produce x and z values of or very close to 0. While the logic makes sense to me, I'm obviously missing something..
If your goal is to rotate a vector parallel to the y axis, why can't you just manually set it using the magnitude of the vector and setting your vector to <0, MAGNITUDE, 0>?
Also, you should know that rotating a vector to be directly pointing +Y or -Y can cause some rotation implementations to break, since they operate according to the "world up" vector, or, <0,1,0>. You can solve this by building your own rotation system and using the "world out" vector <0,0,1> when rotating directly up.
If you have some other purpose for this, fastgraph helped me with building rotation matrices.
It's best to understand the math of what's going on so that you know what to do in the future.