Image Neighbourhood Processing in Haskell - image

Question

I'm new to Haskell, and trying to learn it by thinking in terms of image processing.
So far, I have been stuck thinking about how you would implement a neighbourhood-filtering algorithm in Haskell (or any functional programming language, really).
How would a spatial averaging filter (say 3x3 kernel, 5x5 image) be written functionally? Coming from an entirely imperative background, I can't seem to come up with a way to either structure the data so the solution is elegant, or not do it by iterating through the image matrix, which doesn't seem very declarative.

Answer 1

Working with neighborhoods is easy to do elegantly in a functional language. Operations like convolution with a kernel are higher order functions that can be written in terms of one of the usual tools of functional programming languages - lists.
To write some real, useful code, we'll first play pretend to explain a library.
Pretend
You can think of each image as a function from a coordinate in the image to the value of the data held at that coordinate. This would be defined over all possible coordinates, so it would be useful to pair it with some bounds which tell us where the function is defined. This would suggest a data type like
data Image coordinate value = Image {
lowerBound :: coordinate,
upperBound :: coordinate,
value :: coordinate -> value
}
Haskell has a very similar data type called Array in Data.Array. This data type comes with an additional feature that the value function in Image wouldn't have - it remembers the value for each coordinate so that it never needs to be recomputed. We'll work with Arrays using three functions, which I'll describe in terms of how they'd be defined for Image above. This will help us see that even though we are using the very useful Array type, everything could be written in terms of functions and algebraic data types.
type Array i e = Image i e
bounds gets the bounds of the Array
bounds :: Array i e -> (i, i)
bounds img = (lowerBound img, upperBound img)
The ! looks up a value in the Array
(!) :: Array i e -> i -> e
img ! coordinate = value img coordinate
Finally, makeArray builds an Array
makeArray :: Ix i => (i, i) -> (i -> e) -> Array i e
makeArray (lower, upper) f = Image lower upper f
Ix is a typeclass for things that behave like image coordinates, they have a range. There are instances for most of the base types like Int, Integer, Bool, Char, etc. For example the range of (1, 5) is [1, 2, 3, 4, 5]. There's also an instances for products or tuples of things that themselves have Ix instances; the instance for tuples ranges over all combinations of the ranges of each component. For example, range (('a',1),('c',2)) is
[('a',1),('a',2),
('b',1),('b',2),
('c',1),('c',2)]`
We are only interested in two functions from the Ix typeclass, range :: Ix a => (a, a) -> [a] and inRange :: Ix a => a -> (a, a) -> Bool. inRange quickly checks if a value would be in the result of range.
Reality
In reality, makeArray isn't provided by Data.Array, but we can define it in terms of listArray which constructs an Array from a list of items in the same order as the range of its bounds
import Data.Array
makeArray :: (Ix i) => (i, i) -> (i -> e) -> Array i e
makeArray bounds f = listArray bounds . map f . range $ bounds
When we convolve an array with a kernel, we will compute the neighborhood by adding the coordinates from the kernel to the coordinate we are calculating. The Ix typeclass doesn't require that we can combine two indexes together. There's one candidate typeclass for "things that combine" in base, Monoid, but there aren't instances for Int or Integer or other numbers because there's more than one sensible way to combine them: + and *. To address this, we'll make our own typeclass Offset for things that combine with a new operator called .+.. Usually we don't make typeclasses except for things that have laws. We'll just say that Offset should "work sensibly" with Ix.
class Offset a where
(.+.) :: a -> a -> a
Integers, the default type Haskell uses when you write an integer literal like 9, can be used as offsets.
instance Offset Integer where
(.+.) = (+)
Additionally, pairs or tuples of things that Offset can be combined pairwise.
instance (Offset a, Offset b) => Offset (a, b) where
(x1, y1) .+. (x2, y2) = (x1 .+. x2, y1 .+. y2)
We have one more wrinkle before we write convolve - how will we deal with the edges of the image? I intend to pad them with 0 for simplicity. pad background makes a version of ! that's defined everywhere, outside the bounds of an Array it returns the background.
pad :: Ix i => e -> Array i e -> i -> e
pad background array i =
if inRange (bounds array) i
then array ! i
else background
We're now prepared to write a higher order function for convolve. convolve a b convolves the image b with the kernel a. convolve is higher order because each of its arguments and its result is an Array, which is really a combination of a function ! and its bounds.
convolve :: (Num n, Ix i, Offset i) => Array i n -> Array i n -> Array i n
convolve a b = makeArray (bounds b) f
where
f i = sum . map (g i) . range . bounds $ a
g i o = a ! o * pad 0 b (i .+. o)
To convolve an image b with a kernel a, we make a new image defined over the same bounds as b. Each point in the image can be computed by the function f, which sums the product (*) of the value in the kernel a and the value in the padded image b for each offset o in the range of the bounds of the kernel a.
Example
With the six declarations from the previous section, we can write the example you requested, a spatial averaging filter with a 3x3 kernel applied to a 5x5 image. The kernel a defined below is a 3x3 image that uses one ninth of the value from each of the 9 sampled neighbors. The 5x5 image b is a gradient increasing from 2 in the top left corner to 10 in the bottom right corner.
main = do
let
a = makeArray ((-1, -1), (1, 1)) (const (1.0/9))
b = makeArray ((1,1),(5,5)) (\(x,y) -> fromInteger (x + y))
c = convolve a b
print b
print c
The printed input b is
array ((1,1),(5,5))
[((1,1),2.0),((1,2),3.0),((1,3),4.0),((1,4),5.0),((1,5),6.0)
,((2,1),3.0),((2,2),4.0),((2,3),5.0),((2,4),6.0),((2,5),7.0)
,((3,1),4.0),((3,2),5.0),((3,3),6.0),((3,4),7.0),((3,5),8.0)
,((4,1),5.0),((4,2),6.0),((4,3),7.0),((4,4),8.0),((4,5),9.0)
,((5,1),6.0),((5,2),7.0),((5,3),8.0),((5,4),9.0),((5,5),10.0)]
The convolved output c is
array ((1,1),(5,5))
[((1,1),1.3333333333333333),((1,2),2.333333333333333),((1,3),2.9999999999999996),((1,4),3.6666666666666665),((1,5),2.6666666666666665)
,((2,1),2.333333333333333),((2,2),3.9999999999999996),((2,3),5.0),((2,4),6.0),((2,5),4.333333333333333)
,((3,1),2.9999999999999996),((3,2),5.0),((3,3),6.0),((3,4),7.0),((3,5),5.0)
,((4,1),3.6666666666666665),((4,2),6.0),((4,3),7.0),((4,4),8.0),((4,5),5.666666666666666)
,((5,1),2.6666666666666665),((5,2),4.333333333333333),((5,3),5.0),((5,4),5.666666666666666),((5,5),4.0)]
Depending on the complexity of what you want to do, you might consider using more established libraries, like the oft recommended repa, rather than implementing an image processing kit for yourself.