Incorrect behavior of image undistortion algorithm - algorithm

Question

I'm trying to create a program that receives a photograph of a surface from a certain angle and position, and generates an image of what an isometric projection of the plane would look like. For example, given a photo of a checkerboard
and information about the positioning and properties of the camera, it could reconstruct a section of the undistorted pattern
My approach has been divided into two parts. The first part is to create four rays, coming from the camera, following the four corners of its field of view. I compute where these rays intersect with the plane, to form the quadrangle of the area of the plane that the camera can see, like this:
The second part is to render an isomorphic projection of the plane with the textured quadrangle. I divide the quadrangle into two triangles, then for each pixel on the rendering, I convert the cartesian coordinates into barymetric coordinates relative to each triangle, then convert it back into cartesian coordinates relative to a corresponding triangle that consumes half of the photograph, so that I can sample a color.
(I am aware that this could be done more efficiently with OpenGL, but I would like to not use it for logistical reasons. I am also aware that the quality will be affected by lack of interpolation, that does not matter for this task.)
I am testing the program with some data, but the rendering does not occur as intended. Here is the photograph:
And here is the program output:
I believe that the problem is occurring in the quadrangle rendering, because I have graphed the projected vertices, and they appear to be correct:
I am by no means an expert in computer graphics, so I would very much appreciate if someone had any idea what would cause this problem. Here is the relevant code:
public class ImageProjector {
private static final EquationSystem ground = new EquationSystem(0, 1, 0, 0);
private double fov;
private double aspectRatio;
private vec3d position;
private double xAngle;
private double yAngle;
private double zAngle;
public ImageProjector(double fov, double aspectRatio, vec3d position, double xAngle, double yAngle, double zAngle) {
this.fov = fov;
this.aspectRatio = aspectRatio;
this.position = position;
this.xAngle = xAngle;
this.yAngle = yAngle;
this.zAngle = zAngle;
}
public vec3d[] computeVertices() {
return new vec3d[] {
computeVertex(1, 1),
computeVertex(1, -1),
computeVertex(-1, -1),
computeVertex(-1, 1)
};
}
private vec3d computeVertex(int horizCoef, int vertCoef) {
vec3d p2 = new vec3d(tan(fov / 2) * horizCoef, tan((fov / 2) / aspectRatio) * vertCoef, 1);
p2 = p2.rotateXAxis(xAngle);
p2 = p2.rotateYAxis(yAngle);
p2 = p2.rotateZAxis(zAngle);
if (p2.y > 0) {
throw new RuntimeException("sky is visible to camera: " + p2);
}
p2 = p2.plus(position);
//System.out.println("passing through " + p2);
EquationSystem line = new LineBuilder(position, p2).build();
return new vec3d(line.add(ground).solveVariables());
}
}
public class barypoint {
public barypoint(double u, double v, double w) {
this.u = u;
this.v = v;
this.w = w;
}
public final double u;
public final double v;
public final double w;
public barypoint(vec2d p, vec2d a, vec2d b, vec2d c) {
vec2d v0 = b.minus(a);
vec2d v1 = c.minus(a);
vec2d v2 = p.minus(a);
double d00 = v0.dotProduct(v0);
double d01 = v0.dotProduct(v1);
double d11 = v1.dotProduct(v1);
double d20 = v2.dotProduct(v0);
double d21 = v2.dotProduct(v1);
double denom = d00 * d11 - d01 * d01;
v = (d11 * d20 - d01 * d21) / denom;
w = (d00 * d21 - d01 * d20) / denom;
u = 1.0 - v - w;
}
public barypoint(vec2d p, Triangle triangle) {
this(p, triangle.a, triangle.b, triangle.c);
}
public vec2d toCartesian(vec2d a, vec2d b, vec2d c) {
return new vec2d(
u * a.x + v * b.x + w * c.x,
u * a.y + v * b.y + w * c.y
);
}
public vec2d toCartesian(Triangle triangle) {
return toCartesian(triangle.a, triangle.b, triangle.c);
}
}
public class ImageTransposer {
private BufferedImage source;
private BufferedImage receiver;
public ImageTransposer(BufferedImage source, BufferedImage receiver) {
this.source = source;
this.receiver = receiver;
}
public void transpose(Triangle sourceCoords, Triangle receiverCoords) {
int xMin = (int) Double.min(Double.min(receiverCoords.a.x, receiverCoords.b.x), receiverCoords.c.x);
int xMax = (int) Double.max(Double.max(receiverCoords.a.x, receiverCoords.b.x), receiverCoords.c.x);
int yMin = (int) Double.min(Double.min(receiverCoords.a.y, receiverCoords.b.y), receiverCoords.c.y);
int yMax = (int) Double.max(Double.max(receiverCoords.a.y, receiverCoords.b.y), receiverCoords.c.y);
for (int x = xMin; x <= xMax; x++) {
for (int y = yMin; y <= yMax; y++) {
vec2d p = new vec2d(x, y);
if (receiverCoords.contains(p) && p.x >= 0 && p.y >= 0 && p.x < receiver.getWidth() && y < receiver.getHeight()) {
barypoint bary = new barypoint(p, receiverCoords);
vec2d sp = bary.toCartesian(sourceCoords);
if (sp.x >= 0 && sp.y >= 0 && sp.x < source.getWidth() && sp.y < source.getHeight()) {
receiver.setRGB(x, y, source.getRGB((int) sp.x, (int) sp.y));
}
}
}
}
}
}
public class ProjectionRenderer {
private String imagePath;
private BufferedImage mat;
private vec3d[] vertices;
private vec2d pos;
private double scale;
private int width;
private int height;
public boolean error = false;
public ProjectionRenderer(String image, BufferedImage mat, vec3d[] vertices, vec3d pos, double scale, int width, int height) {
this.imagePath = image;
this.mat = mat;
this.vertices = vertices;
this.pos = new vec2d(pos.x, pos.z);
this.scale = scale;
this.width = width;
this.height = height;
}
public void run() {
try {
BufferedImage image = ImageIO.read(new File(imagePath));
vec2d[] transVerts = Arrays.stream(vertices)
.map(v -> new vec2d(v.x, v.z))
.map(v -> v.minus(pos))
.map(v -> v.multiply(scale))
.map(v -> v.plus(new vec2d(mat.getWidth() / 2, mat.getHeight() / 2)))
// this fixes the image being upside down
.map(v -> new vec2d(v.x, mat.getHeight() / 2 + (mat.getHeight() / 2 - v.y)))
.toArray(vec2d[]::new);
System.out.println(Arrays.toString(transVerts));
Triangle sourceTri1 = new Triangle(
new vec2d(0, 0),
new vec2d(image.getWidth(), 0),
new vec2d(0, image.getHeight())
);
Triangle sourceTri2 = new Triangle(
new vec2d(image.getWidth(), image.getHeight()),
new vec2d(0, image.getHeight()),
new vec2d(image.getWidth(), 0)
);
Triangle destTri1 = new Triangle(
transVerts[3],
transVerts[0],
transVerts[2]
);
Triangle destTri2 = new Triangle(
transVerts[1],
transVerts[2],
transVerts[0]
);
ImageTransposer transposer = new ImageTransposer(image, mat);
System.out.println("transposing " + sourceTri1 + " -> " + destTri1);
transposer.transpose(sourceTri1, destTri1);
System.out.println("transposing " + sourceTri2 + " -> " + destTri2);
transposer.transpose(sourceTri2, destTri2);
} catch (IOException e) {
e.printStackTrace();
error = true;
}
}
}

Answer 1

The reason it's not working is because your transpose function works entirely with 2D co-ordinates, therefore it cannot compensate for the image distortion resulting from 3D perspective. You have effectively implemented a 2D affine transformation. Parallel lines remain parallel, which they do not under a 3D perspective transform. If you draw a straight line between two points on your triangle, you can linearly interpolate between them by linearly interpolating the barycentric co-ordinates, and vice versa.
To take Z into account, you can keep the barycentric co-ordinate approach, but provide a Z co-ordinate for each point in sourceCoords. The trick is to interpolate between 1/Z values (which can be linearly interpolated in a perspective image) instead of interpolating Z itself. So instead of interpolating what are effectively the texture co-ordinates for each point, interpolate the texture co-ordinate divided by Z, along with inverse Z, and interpolate all of those using your barycentric system. Then divide by inverse Z before doing your texture lookup to get texture co-ordinates back.
You could do that like this (assume a b c contain an extra z co-ordinate giving distance from camera):
public vec3d toCartesianInvZ(vec3d a, vec3d b, vec3d c) {
// put some asserts in to check for z = 0 to avoid div by zero
return new vec3d(
u * a.x/a.z + v * b.x/b.z + w * c.x/c.z,
u * a.y/a.z + v * b.y/b.z + w * c.y/c.z,
u * 1/a.z + v * 1/b.z + w * 1/c.z
);
}
(You could obviously speed up/simplify this by pre-computing all those divides and storing in sourceCoords, and just doing regular barycentric interpolation in 3D)
Then after you call it in transpose, divide by inv Z to get the texture co-ords back:
vec3d spInvZ = bary.toCartesianInvZ(sourceCoords);
vec2d sp = new vec2d(spInvZ.x / spInvZ.z, spInvZ.y / spInvZ.z);
etc. The Z co-ordinate that you need is the distance of the point in 3D space from the camera position, in the direction the camera is pointing. You can compute it with a dot product if you aren't getting it some other way:
float z = point.subtract(camera_pos).dot(camera_direction);
etc