Julia equivalent of Python multiprocessing.Pool.map - multiprocessing

Question

My multi processing needs are very simple: I work in machine learning, and I sometimes need to evaluate an algorithm in multiple datasets, or multiple algorithms in a dataset, or some such. I just need to run a function with some arguments and get a number.
I need no RPC, shared data, nothing.
In Julia, I am getting an error with the following code:
type Model
param
end
# 1. I have several algorithms/models
models = [Model(i) for i in 1:50]
# 2. I have one dataset
X = rand(50, 5)
# 3. I want to paralelize this function
#everywhere function transform(m)
sum(X .* m.param)
end
addprocs(3)
println(pmap(transform, models))
I keep getting errors such as,
ERROR: LoadError: On worker 2:
UndefVarError: #transform not defined
Also, is there a way to avoid having to place #everywhere everywhere? Can I just tell that all variables should be copied over to the workers when they are created (as is done in Python multiprocessing)?
My typical code looks obviously much more complicated than this, with models ranging several files.
For reference, this is what I would do in Python:
import numpy as np
import time
# 1. I have several algorithms/models
class Model:
def __init__(self, param):
self.param = param
models = [Model(i) for i in range(1,51)]
# 2. I have one dataset
X = np.random.random((50, 5))
# 3. I want to paralelize this function
def transform(m):
return np.sum(X * m.param)
import multiprocessing
pool = multiprocessing.Pool(4)
print(pool.map(transform, models))

Answer 1

Core issues is you need to add the processes before you attempt to define things on them.
addprocs should always be the first thing you do, before using even (see below).
This is why it is often done with the -p flag when you start julia.
Or with a ---machinefile <file> or with a -L <file>
#everywhere exectutes the code on all processes the currently exist.
i.e. process added after the #everywhere do not have the code executed on them.
Also you missed a few #everywheres.
addprocs(3)
#everywhere type Model
param
end
# 1. I have several algorithms/models
models = [Model(i) for i in 1:50]
# 2. I have one dataset
#everywhere X = rand(50, 5)
# 3. I want to paralelize this function
#everywhere function transform(m)
sum(X .* m.param)
end
println(pmap(transform, models))
Alternatives with fewer #everywheres.
use a block to send a whole block of code #everywhere
addprocs(3)
#everywhere begin
type Model
param
end
X = rand(50, 5)
function transform(m)
sum(X .* m.param)
end
end
models = [Model(i) for i in 1:50]
println(pmap(transform, models))
Use local variables
Local variables (including functions), are sent as required.
though this doesn't help for types.
addprocs(3)
#everywhere type Model
param
end
function main()
X = rand(50, 5)
models = [Model(i) for i in 1:50]
function transform(m)
sum(X .* m.param)
end
println(pmap(transform, models))
end
main()
Use modules
When you using Foo the module Foo is loaded on all processes.
But not brought into scope.
It is a bit weird and counter intuitive.
So much so that I can't conjure a working example of it.
but someone else might.