How do convolution matrices work? - image

Question

How do those matrices work? Do I need to multiple every single pixel? How about the upperleft, upperright, bottomleft and bottomleft pixels where there's no surrounding pixel? And does the matrix work from left to right and from up to bottom or from up to bottom first and then left to right?
Why does this kernel (Edge enhance) : http://i.stack.imgur.com/d755G.png
turns into this image: http://i.stack.imgur.com/NRdkK.jpg

Answer 1

The Convolution filter is applied to every single pixel.
On the edges there are a few things you can do (all leave a type of border or shrink the image):
skip the edges and crop 1 pixel from the edge of the image
substitute 0 or 255 for any of the pixels that are out of bounds for the image
use a cubic spline (or other interpolation method) between 0 (or 255) and the value of the images edge pixel to come up with a substitute.
The order you apply the convolution does not matter (upper right to bottom left is most common) you should get the same results no matter the order.
However, a common mistake when applying a convolution matrix is to overwrite the current pixel you are examining with the new value. This will affect the value you come up with for the pixel next to the current one. A better method would be to create a buffer to hold the computed values, so that previous applications of the convolution filter do not affect current application of the matrix.
From your example images it is hard to tell why the filter applied creates the black and white version without seeing the original image.

Answer 2

Below is a step by step example of applying a convolution kernel to an image (1D for simplicity).
As for the edge enhancement kernel in your post, notice the +1 next to the -1. Think about what that will do. If the region is constant the two pixel under the +/-1 will add to zero (black). If the two pixels are different they will have a non-zero value. So what you are seeing is that pixels next to each other that are different get highlighted, while ones that are the same get set to black. The bigger the difference the brighter (more white) the pixel in the filtered image.

Answer 3

Yes, you multiply every pixel, with that matrix. The traditional method is to find the relevant pixels relative to the pixel being convoluted, multiple the factors, and average it out. So a 3x3 blur of:
1, 1, 1,
1, 1, 1,
1, 1, 1
This matrix, means you take the relevant values of the various components and multiply them. Then divide by the number of elements. So you would get that 3 by 3 box, add up all the red values then divide by 9. You'd get the 3 by 3 box, add up all the green values then divide by 9. You'd get the 3 by 3 box, add up all the blue values then divide by 9.
This means a couple things. First, you need a second giant chunk of memory to perform this operation. And you do every pixel you can.
However, that's only for the traditional method and the traditional method is actually needlessly convoluted (get it?). If you return the results in a corner. You never actually need any additional memory and always do the entire operation within the memory footprint you started with.
public static void convolve(int[] pixels, int offset, int stride, int x, int y, int width, int height, int[][] matrix, int parts) {
int index = offset + x + (y*stride);
for (int j = 0; j < height; j++, index += stride) {
for (int k = 0; k < width; k++) {
int pos = index + k;
pixels[pos] = convolve(pixels,stride,pos, matrix, parts);
}
}
}
private static int crimp(int color) {
return (color >= 0xFF) ? 0xFF : (color < 0) ? 0 : color;
}
private static int convolve(int[] pixels, int stride, int index, int[][] matrix, int parts) {
int redSum = 0;
int greenSum = 0;
int blueSum = 0;
int pixel, factor;
for (int j = 0, m = matrix.length; j < m; j++, index+=stride) {
for (int k = 0, n = matrix[j].length; k < n; k++) {
pixel = pixels[index + k];
factor = matrix[j][k];
redSum += factor * ((pixel >> 16) & 0xFF);
greenSum += factor * ((pixel >> 8) & 0xFF);
blueSum += factor * ((pixel) & 0xFF);
}
}
return 0xFF000000 | ((crimp(redSum / parts) << 16) | (crimp(greenSum / parts) << 8) | (crimp(blueSum / parts)));
}
With the kernel traditionally returning the value to the center most pixel. This allows the image to blur around the edges but more or less remain where it started. This seemed like a good idea but it's actually problematic. The correct way to do it is to have the results pixel in the upper-left corner. Then you can simply, and with no extra memory, just iterate the the entire image with a scanline, going one pixel at a time and returning the value, without causing errors. The bulk of the color weight is shifted up and left by one pixel. But, it's one pixel, and you can shift it back down and to the left if you iterate backwards with a result pixel in the bottom-right. Though this might be trouble for the cache hits.
However, a lot of modern architecture have GPUs now, so the entire image can be done simultaneously. Making it a kind of moot point. But, it is strange that one of the most important algorithm in graphics is weird in requiring this, as that makes the easiest way to do the operation impossible, and a memory hog.
So that people like Matt on this question say things like "However, a common mistake when applying a convolution matrix is to overwrite the current pixel you are examining with the new value." -- Really this is the correct way to do it, the error is writing the result pixel to the center rather than the upper left corner. Because unlike the upper-left corner, you will need the center pixel again. You won't ever need the upper-left corner again (assuming you are iterating left->right, top->bottom), and so it's safe to store your value there.
"This will affect the value you come up with for the pixel next to the current one." -- If you wrote it to the upper left corner as you processed it as a scan, you would overwrite data that you do not ever need again. Using a bunch of extra memory isn't a better solution.
As such, here's likely the fastest Java blur you'd ever see.
private static void applyBlur(int[] pixels, int stride) {
int v0, v1, v2, r, g, b;
int pos;
pos = 0;
try {
while (true) {
v0 = pixels[pos];
v1 = pixels[pos+1];
v2 = pixels[pos+2];
r = ((v0 >> 16) & 0xFF) + ((v1 >> 16) & 0xFF) + ((v2 >> 16) & 0xFF);
g = ((v0 >> 8 ) & 0xFF) + ((v1 >> 8) & 0xFF) + ((v2 >> 8) & 0xFF);
b = ((v0 ) & 0xFF) + ((v1 ) & 0xFF) + ((v2 ) & 0xFF);
r/=3;
g/=3;
b/=3;
pixels[pos++] = r << 16 | g << 8 | b;
}
}
catch (ArrayIndexOutOfBoundsException e) { }
pos = 0;
try {
while (true) {
v0 = pixels[pos];
v1 = pixels[pos+stride];
v2 = pixels[pos+stride+stride];
r = ((v0 >> 16) & 0xFF) + ((v1 >> 16) & 0xFF) + ((v2 >> 16) & 0xFF);
g = ((v0 >> 8 ) & 0xFF) + ((v1 >> 8) & 0xFF) + ((v2 >> 8) & 0xFF);
b = ((v0 ) & 0xFF) + ((v1 ) & 0xFF) + ((v2 ) & 0xFF);
r/=3;
g/=3;
b/=3;
pixels[pos++] = r << 16 | g << 8 | b;
}
}
catch (ArrayIndexOutOfBoundsException e) { }
}