I have a rather vague understanding of how rasterization is supposed to work.
So I totally understand how vertices make up a 3d image. I also ventured into model to world projection and even though I don't understand the math behind it ( I use helper libraries to multiply the matrices and have a chart denoting how to apply different transformations: rotate, scale, translate, etc).
So it's very easy for me to build some 3d model using blender and apply that logic to build a world matrix for each object.
But i've hit a brick wall trying to envision how to camera matrix is supposed to "look at" a specific cluster of vertices and what exactly happens to the object's world coordinates after the camera matrix is applied to the world matrix? and what does a camera matrix look like and how does the camera's "view axis" affect it's matrix (the camera could be looking at the z,x, y axis respectively)
I've managed to render a couple 3d objects with various rendering engines (openGL, XNA, etc) but most of it was due to having followed some guide on the internet or trying to interpret what some guy on youtube is trying to teach, and i'm still struggling trying to get an "intuitive" sense on how matrices are supposed to work as per camera parameters and how the camera is supposed to alter the object's world matrix
There are 5 steps in going from "world space" (Wx,Wy,Wz) to "screen space" (Sx,Sy): View, Clipping, Projection, Perspective Divide, Viewport. This is described pretty well here but some details are glossed over. I will try to explain the steps conceptually.
Imagine you have some vertices (what we want to render), a camera (with a position and orientation - which direction it is pointing), and a screen (a rectangular grid of WIDTHxHEIGHT pixels).
The Model Matrix I think you already understand: it scales, rotates, and translates each vertex into world coordinates (Wx,Wy,Wz,1.0). The last "1.0" (sometimes called the w component) allows us to represent translation and projection (as well as scaling and rotation) as a single 4x4 matrix.
The View Matrix (aka camera matrix) moves all the vertices to the point of view of the camera. I think of it as working in 2 steps: First it translates the entire scene (all vertices including the camera) such that in the new coordinate system the camera is at the origin. Second it rotates the entire scene such that the camera is looking from the origin in the direction of the -Z axis. There is a good description of this here. (Mathematically the rotation happens first, but I find it easier to visualize if I do the translation first.) At this point each vertex is in View coordinates (Vx,Vy,Vz,1.0). A good way to visualize this is to imagine the entire scene is embedded in ice; grab the block of ice and move it so the camera is at the origin pointing along the -z axis (and all the other objects in the world move along with the ice they are embedded in).
Next, the projection matrix encodes what kind of lens (wide angle vs telephoto) the camera has; in other words how much of the world will be visible on the screen. This is described well here but here is how it is calculated:
[ near/width ][ 0 ][ 0 ][ 0 ]
[ 0 ][ near/height ][ 0 ][ 0 ]
[ 0 ][ 0 ][(far+near)/(far-near) ][ 1 ]
[ 0 ][ 0 ][-(2*near*far)/(far-near)][ 0 ]
near = near plane distance (everything closer to the camera than this is clipped).
far = far plane distance (everything farther from the camera than this is clipped).
width = the widest object we can see if it is at the near plane.
height = the tallest object we can see if it is at the near plane.
. It results in "clip coordinates" (Cx,Cy,Cz,Cw=Vz). Note that the viewspace z coordinate (Vz) ends up in the w coordinate of the clip coordinates (Cw) (more on this below). This matrix stretches the world so that the camera's field of view is now 45 degrees up,down,left, and right. In other words, in this coordinate system if you look from the origin (camera position) straight along the -z axis (direction the camera is pointing) you will see what is in the center of the screen, and if you rotate your head up {down,left,right} you will see what will be at the top {bottom,left,right} of the screen. You can visualize this as a pyramid shape where the camera is at the top of the pyramid and the camera is looking straight down inside the pyramid. (This shape is called a "frustum" once you clip the top and bottom of the pyramid off with the near and far plane - see next paragraph.) The Cz value calculation makes vertices at the near plane have Cz=-Cw and vertices at the far plane have Cz=Cw
Clipping takes place in clip coordinates (which is why they are called that). Clipping means you take some scissors and clip away anything that is outside that pyramid shape. You also clip everything that is too close to the camera (the "near plane") and everything that is too far away from the camera (the "far plane"). See here for details.
Next comes the perspective divide. Remember that Cw == Vz? This is the distance from the camera to the vertex along the z axiz (the direction the camera is pointing). We divide each component by this Cw value to get Normalized Projection Coordinates (NPC) (Nx=Cx/Cw, Ny=Cy/Cw, Nz=Cz/Cw, Nw=Cw/Cw=1.0). All these values (Nx, Ny and Nz) will be between -1 and 1 because we clipped away anything where Cx > Cw or Cx < -Cw or Cy > Cw or Cy < -Cw or Cz > Cw or Cz < -Cw. Again see here for lots of details on this. The perspective divide is what makes things that are farther away appear smaller. The farther away from the camera something is, the larger the Cw (Vz) is, and the more its X and Y coordinate will be reduced when we divide.
The final step is the viewport transform. Nx Ny and Nz (each ranging from -1 to 1) are converted to pixel coordinates. For example Nx=-1 is at the left of the screen and Nx=1 is at the right of the screen, so we get Sx = (Nx * WIDTH/2) + (WIDTH/2) or equivalently Sx = (Nx+1) * WIDTH. Similar for Sy. You can think of Sz as the value that will be used in a depth buffer, so it needs to range from 0 for vertices at the near plane (Vz=near) to the maximum value that the depth buffer can hold (e.g. 2^24= 16777216 for a 24 bit z buffer) for vertices at the far plane (Vz=far).
The "camera matrix" as you called it sounds like a combination of two matrices: the view matrix and the projection matrix. It's possible you're only talking about one of these, but it's not clear.
View matrix: The view matrix is the inverse of what the camera's model matrix would be if you drew it in the world. In order to draw different camera angles, we actually move the entire world in the opposite direction - so there is only one camera angle.
Usually in OpenGL, the camera "really" stays at (0,0,0) and looks along the Z axis in the positive direction (towards 0,0,+∞). You can apply a rotation to the projection matrix to get a different direction, but why would you? Do all the rotation in the view matrix and your life is simpler.
So if you want your camera to be at (0,3,0) for example, instead of moving the camera up 3 units, we leave it at (0,0,0) and move the entire world down 3 units. If you want it to rotate 90 degrees, we actually rotate the world 90 degrees in the opposite direction. The world doesn't mind - it's just numbers in a computer - it doesn't get dizzy.
We only do this when rendering. All of the game physics calculations, for example, aren't done in the rotated world. The coordinates of the stuff in the world don't get changed when we rotate the camera - except inside the rendering system. Usually, we tell the GPU the normal world coordinates of the objects, and we get the GPU to move and rotate them for us, using a matrix.
Projection matrix: You know the view frustum? This shape you've probably seen before: (credit)
Everything inside the cut-off pyramid shape (frustum) is displayed on the screen. You know this.
Except the computer doesn't actually render in a frustum. It renders a cube. The view matrix transforms the frustum into a cube.
If you're familiar with linear algebra, you may notice that a 3D matrix can't make a cube into a frustum. That's what the 4th coordinate (w) is for. After this calculation, the x, y and z coordinates are all divided by w. By using a view matrix that makes w depend on z, the coordinates of far-away points get divided by a larger number, so they get pushed towards the middle of the screen - that's how a cube is able to turn into a frustum.
You don't have to have a frustum - that's what you get with a perspective projection. You can also use an orthographic projection, which turns a cube into a cube, by not changing w.
Unless you want to do a bunch of math yourself, I'd recommend you just use the library functions to generate projection matrices.
If you're multiplying vertices by several matrices in a row it's more efficient to combine them into one matrix, and then multiply the vertices by the combined matrix - hence you will often see MVP, MV and VP matrices used. (M = model matrix - I think it's the same thing you called a world matrix)
Related
So, for anyone familiar with Google Maps, when you zoom, it does it around the cursor.
That is to say, the matrix transformation for such a zoom is as simple as:
TST^{-1}*x
Where T is the translation matrix representing the point of focus, S the scale matrix and x is any arbitrary point on the plane.
Now, I want to produce a similar effect with a spherical camera, think sketchfab.
When you zoom in and out, the camera needs to be translated so as to give a similar effect as the 2D zooming in Maps. To be more precise, given a fully composed MVP matrix, there exists a set of parallel planes that are parallel to the camera plane. Among those there exists a unique plane P that also contains the center of the current spherical camera.
Given that plane, there exists a point x, that is the unprojection of the current cursor position onto the camera plane.
If the center of the spherical camera is c then the direction from c to x is d = x - c.
And here's where my challenge comes. Zooming is implemented as just offsetting the camera radially from the center, given a change in zoom Delta, I need to find the translation vector u, colinear with d, that moves the center of the camera towards x, such that I get a similar visual effect as zooming in google maps.
Since I know this is a bit hard to parse I tried to make a diagram:
TL;DR
I want to offset a spherical camera towards the cursor when I zoom, how do i pick my translation vector?
I am working on a Perspective camera. The constructor must be:
PerspectiveCamera::PerspectiveCamera(Vec3f ¢er, Vec3f &direction, Vec3f &up, float angle)
This is construction different from most others, as it lacks near and far clipping planes. I know what to with center, direction, and up -- the standard look at algorithm.
We can construct the view matrix and translate matrix accordingly:
Thus, the viewing transformation is:
For an orthographic camera (which is working correctly for me), the inverse transformation is used to go from screen space to world space. The camera coordinates go from (-1,-1,0) --> (1,1,0) in screen space.
For perspective transformation, only the field of view is given. The Wikipedia 3D projection article gives a perspective projection matrix using the field of view angle and assuming camera coordinates go from (-1,-1) --> (1,1):
In my code, (ex,ey,ez) are the camera coordinates that go from (-1,-1, ez) --> (1,1, ez). Note that the 1 in (3,3) spot of K isn't in the Wikipedia article -- I put it in to make the matrix invertible. So that may be a problem.
But anyways, for perspective projection, I used this transformation:
K inverse is multiplied with p to make the canonical view volume to a view frustum, and the result of that is multiplied with M inverse to move into world coordinates.
I get the wrong results. The correct output is:
My output looks like this:
Am I using the right algorithm for perspective projection given my constraint (no near and far plane inputs)???
Just in case somebody else runs into this issue, the method presented in the question is not proper way to create a viewing frustum. The perspective matrix (K) is for projecting the far plane onto the near plane, and we don't have those planes in this case
To create a frustum, do the inverse transformation on (x,y,ez) [as opposed to (x,y,0) for orthographic projection). Find a new direction by subtracting the transformed point from the center of projection. Shoot the ray.
I'm trying to write a simple voxel raycaster as a learning exercise. This is purely CPU based for now until I figure out how things work exactly -- fow now, OpenGL is just (ab)used to blit the generated bitmap to the screen as often as possible.
Now I have gotten to the point where a perspective-projection camera can move through the world and I can render (mostly, minus some artifacts that need investigation) perspective-correct 3-dimensional views of the "world", which is basically empty but contains a voxel cube of the Stanford Bunny.
So I have a camera that I can move up and down, strafe left and right and "walk forward/backward" -- all axis-aligned so far, no camera rotations. Herein lies my problem.
Screenshots: (1) raycasting voxels while... ...(2) the camera remains... ...(3) strictly axis-aligned.
Now I have for a few days been trying to get rotation to work. The basic logic and theory behind matrices and 3D rotations, in theory, is very clear to me. Yet I have only ever achieved a "2.5 rendering" when the camera rotates... fish-eyey, bit like in Google Streetview: even though I have a volumetric world representation, it seems --no matter what I try-- like I would first create a rendering from the "front view", then rotate that flat rendering according to camera rotation. Needless to say, I'm by now aware that rotating rays is not particularly necessary and error-prone.
Still, in my most recent setup, with the most simplified raycast ray-position-and-direction algorithm possible, my rotation still produces the same fish-eyey flat-render-rotated style looks:
camera "rotated to the right by 39 degrees" -- note how the blue-shaded left-hand side of the cube from screen #2 is not visible in this rotation, yet by now "it really should"!
Now of course I'm aware of this: in a simple axis-aligned-no-rotation-setup like I had in the beginning, the ray simply traverses in small steps the positive z-direction, diverging to the left or right and top or bottom only depending on pixel position and projection matrix. As I "rotate the camera to the right or left" -- ie I rotate it around the Y-axis -- those very steps should be simply transformed by the proper rotation matrix, right? So for forward-traversal the Z-step gets a bit smaller the more the cam rotates, offset by an "increase" in the X-step. Yet for the pixel-position-based horizontal+vertical-divergence, increasing fractions of the x-step need to be "added" to the z-step. Somehow, none of my many matrices that I experimented with, nor my experiments with matrix-less hardcoded verbose sin/cos calculations really get this part right.
Here's my basic per-ray pre-traversal algorithm -- syntax in Go, but take it as pseudocode:
fx and fy: pixel positions x and y
rayPos: vec3 for the ray starting position in world-space (calculated as below)
rayDir: vec3 for the xyz-steps to be added to rayPos in each step during ray traversal
rayStep: a temporary vec3
camPos: vec3 for the camera position in world space
camRad: vec3 for camera rotation in radians
pmat: typical perspective projection matrix
The algorithm / pseudocode:
// 1: rayPos is for now "this pixel, as a vector on the view plane in 3d, at The Origin"
rayPos.X, rayPos.Y, rayPos.Z = ((fx / width) - 0.5), ((fy / height) - 0.5), 0
// 2: rotate around Y axis depending on cam rotation. No prob since view plane still at Origin 0,0,0
rayPos.MultMat(num.NewDmat4RotationY(camRad.Y))
// 3: a temp vec3. planeDist is -0.15 or some such -- fov-based dist of view plane from eye and also the non-normalized, "in axis-aligned world" traversal step size "forward into the screen"
rayStep.X, rayStep.Y, rayStep.Z = 0, 0, planeDist
// 4: rotate this too -- 0,zstep should become some meaningful xzstep,xzstep
rayStep.MultMat(num.NewDmat4RotationY(CamRad.Y))
// set up direction vector from still-origin-based-ray-position-off-rotated-view-plane plus rotated-zstep-vector
rayDir.X, rayDir.Y, rayDir.Z = -rayPos.X - me.rayStep.X, -rayPos.Y, rayPos.Z + rayStep.Z
// perspective projection
rayDir.Normalize()
rayDir.MultMat(pmat)
// before traversal, the ray starting position has to be transformed from origin-relative to campos-relative
rayPos.Add(camPos)
I'm skipping the traversal and sampling parts -- as per screens #1 through #3, those are "basically mostly correct" (though not pretty) -- when axis-aligned / unrotated.
It's a lot easier if you picture the system as a pinhole camera rather than anything else. Instead of shooting rays from the surface of a rectangle representing your image, shoot the rays from a point, through the rectangle that will be your image plane, into the scene. All the primary rays should have the same point of origin, only with slightly different directions. The directions are determined using basic trig by which pixel in the image plane you want them to go through. To make the simplest example, let's imagine your point is at the camera, and your image plane is one unit along the z axis, and two units tall and wide. That way, the pixel at the upper-left corner wants to go from (0,0,0) through (-1, -1, 1). Normalize (-1, -1, 1) to get the direction. (You don't actually need to normalize the direction just to do ray intersection, but if you decide not to, remember that your directions are non-normalized before you try to compute the distance the ray has travelled or anything like that.) For every other pixel, compute the point on the plane it wants to go through the way you've already been doing, by dividing the size of the plane by the number of pixels, in each direction.
Then, and this is the most important thing, don't try to do a perspective projection. That's necessary for scan-conversion techniques, to map every vertex to a point on the screen, but in ray-tracing, your rays accomplish that just by spreading out from one point into space. The direction from your start point (camera position, the origin in this example), through your image plane, is exactly the direction you need to trace with. If you were to want an orthographic projection instead (and you almost never want this), you'd accomplish this by having the direction be the same for all the rays, and the starting positions vary across the image plane.
If you do that, you'll have a good starting point. Then you can try again to add camera rotation, either by rotating the image plane about the origin before you iterate over it to compute ray directions, or by rotating the ray directions directly. There's nothing wrong with rotating directions directly! When you bear in mind that a direction is just the position your ray goes through if it starts from the origin, it's easy to see that rotating the direction, and rotating the point it goes through, do exactly the same thing.
I have a stack of images (about 180 of them) and there are 2 stars (just basic annotations) on every single image. Hence, the position (x,y) of the two stars are provided initially. The dimensions of all these images are fixed and constant.
The 'distance' between the image is about 1o with the origin to be the center (width/2, height/2) of every single 2D image. Note that, if this is plotted out and interpolated nicely, the stars would actually form a ring of an irregular shape.
The dotted red circle and dotted purple circle are there to give a stronger scent of a 3D space and the arrangement of the 2D images (like a fan). It also indicates that each slice is about 1o apart.
With the provided (x,y) that appeared in the 2D image, how do you get the corresponding (x,y,z) in the 3d space knowing that each image is about 1o apart?
I know that MATLAB had 3D plotting capabilities, how should I go about implementing the solution to the above scenario? (Unfortunately, I have very little experience plotting 3D with MATLAB)
SOLUTION
Based on the accepted answer, I looked up a bit further: spherical coordinate system. Based on the computation of phi, rho and theta, I could reconstruct the ring without problems. Hopefully this helps anyone with similar problems.
I have also documented the solution here. I hope it helps someone out there, too:
http://gray-suit.blogspot.com/2011/07/spherical-coordinate-system.html
I believe the y coordinate stays as is for 3D, so we can treat this as converting 2D x and image angle to an x and z when viewed top down.
The 2D x coordinate is the distance from the origin in 3D space (viewed top down). The image angle is the angle the point makes with respect to the x axis in 3D space (viewed top down). So the x coordinate (distance from orign) and the image angle (angle viewed top down) makes up the x and z coordinates in 3D space (or x and y if viewed top down).
That is a polar coordinate.
Read how to convert from polar to cartesian coordinates to get your 3D x and z coordinates.
I'm not great at maths either, here's my go:
3D coords = (2Dx * cos(imageangle), 2Dy, 2Dx * sin(imageangle))
Given the 2D coordinates (x,y) just add the angle A as a third coordinate: (x,y,A). Then you have 3D.
If you want to have the Anotations move on a circle of radius r in 3D you can just calculate:
you can use (r*cos(phi),r*sin(phi),0) which draws a circle in the XY-plane and rotate it with a 3x3 rotation matrix into the orientation you need.
It is not clear from you question around which axis your rotation is taking place. However, my answer holds for a general rotation axis.
First, put your points in a 3D space, lying on the X-Y plane. This means the points have a 0 z-coordinate. Then, apply a 3D rotation of the desired angle around the desired axis - in your example, it is a one degree rotation. You could calculate the transformation matrix yourself (should not be too hard, google "3D rotation matrix" or similar keywords). However, MATLAB makes it easier, using the viewmtx function, which gives you a 4x4 rotational matrix. The extra (fourth) dimension is dependent on the projection you specify (it acts like a scaling coefficient), but in order to make things simple, I will let MATLAB use its default projection - you can read about it in MATLAB documentation.
So, to make the plot clearer, I assume four points which are the vertices of a square lying on the x-y plane (A(1,1), B(1,-1), C(-1,-1), D(1,-1)).
az = 0; % Angle (degrees) of rotation around the z axis, measured from -y axis.
el = 90; % Angle (degrees) of rotation around the y' axis (the ' indicates axes after the first rotation).
x = [1,-1, -1, 1,1]; y = [1, 1, -1, -1,1]; z = [0,0, 0, 0,0]; % A square lying on the X-Y plane.
[m,n] = size(x);
x4d = [x(:),y(:),z(:),ones(m*n,1)]'; % The 4D version of the points.
figure
for el = 90 : -1 :0 % Start from 90 for viewing directly above the X-Y plane.
T = viewmtx(az, el);
x2d = T * x4d; % Rotated version of points.
plot3 (x2d(1,:), x2d(2,:),x2d(3,:),'-*'); % Plot the rotated points in 3D space.
grid
xlim ([-2,2]);
ylim ([-2,2]);
zlim([-2,2]);
pause(0.1)
end
If you can describe your observation of a real physical system (like a binary star system) with a model, you can use particle filters.
Those filters were developed to locate a ship on the sea, when only one observation direction was available. One tracks the ship and estimates where it is and how fast it moves, the longer one follows, the better the estimates become.
I'm building a simple 3D drag and drop interface in processing, and want to detect when the mouse rolls over an object. I would imagine that I need to do some matrix translations to the 3D model coordinates to get them into screen space and so on ...
I have a simple version of this working, the problem is that when camera is moved around the scene the coordinates I get go haywire.
So how do I translate the tile coordinates into screen space (since the screenX & screenY aren't working properly)?
UPDATE:
I eventually found two examples from the Processing site on how to do this. Thanks to villintehaspam.
http://processing.org/hacks/hacks:picking
This problem is called picking. Search for mouse picking and you get lots and lots of hits.
Basic theory is this:
Get x,y coords from the mouse click.
Convert these to x,y,z coordinates in eye coordinates (i.e -1 <= x <= 1, -1 <= y <= 1, z=near/far clip distance, if you have a normal projection).
Transform these coordinates by the inverse of the projection matrix to get world coordinates.
You now have a ray from the camera position, with the direction towards the world coordinates you just got.
Make a ray-object intersection test with the objects you want to consider. Choose the object that intersects the ray that is closest to the ray origin (camera position).