Alternative use patterns for python multiprocessing avoiding proliferation of global state? - multiprocessing

Question

This (enormously simplified example) works fine (Python 2.6.6, Debian Squeeze):
from multiprocessing import Pool
import numpy as np
src=None
def process(row):
return np.sum(src[row])
def main():
global src
src=np.ones((100,100))
pool=Pool(processes=16)
rows=pool.map(process,range(100))
print rows
if __name__ == "__main__":
main()
however, after years of being taught global state bad!!!, all my instincts are telling me I really really would rather be writing something closer to:
from multiprocessing import Pool
import numpy as np
def main():
src=np.ones((100,100))
def process(row):
return np.sum(src[row])
pool=Pool(processes=16)
rows=pool.map(process,range(100))
print rows
if __name__ == "__main__":
main()
but of course that doesn't work (hangs up unable to pickle something).
The example here is trivial, but by the time you add multiple "process" functions, and each of those is dependent on multiple additional inputs... well it all becomes a bit reminiscent of something written in BASIC 30 years ago. Trying to use classes to at least aggregate the state with the appropriate functions seems an obvious solution, but doesn't seem to be that easy in practice.
Is there some recommended pattern or style for using multiprocessing.pool which will avoid the proliferation of global state to support each function I want to parallel map over ?
How do experienced "multiprocessing pros" deal with this ?
Update: Note that I'm actually interested in processing much bigger arrays, so variations on the above which pickle src each call/iteration aren't nearly as good as ones which fork it into the pool's worker processes.

Answer 1

You could always pass a callable object like this, then the object can containe the shared state:
from multiprocessing import Pool
import numpy as np
class RowProcessor(object):
def __init__(self, src):
self.__src = src
def __call__(self, row):
return np.sum(self.__src[row])
def main():
src=np.ones((100,100))
p = RowProcessor(src)
pool=Pool(processes=16)
rows = pool.map(p, range(100))
print rows
if __name__ == "__main__":
main()