Replace pickle with dill and pymongo call - parallel-processing

Question

I finally understood example how to replace pickle with dill from the following discussion: pickle-dill.
For example, the following code worked for me
import os
import dill
import multiprocessing
def run_dill_encoded(what):
fun, args = dill.loads(what)
return fun(*args)
def apply_async(pool, fun, args):
return pool.apply_async(run_dill_encoded, (dill.dumps((fun, args)),))
if __name__ == '__main__':
pool = multiprocessing.Pool(5)
results = [apply_async(pool, lambda x: x*x, args=(x,)) for x in range(1,7)]
output = [p.get() for p in results]
print(output)
I tried to apply the same philosophy to pymongo. The following code
import os
import dill
import multiprocessing
import pymongo
def run_dill_encoded(what):
fun, args = dill.loads(what)
return fun(*args)
def apply_async(pool, fun, args):
return pool.apply_async(run_dill_encoded, (dill.dumps((fun, args)),))
def write_to_db(value_to_insert):
client = pymongo.MongoClient('localhost', 27017)
db = client['somedb']
collection = db['somecollection']
result = collection.insert_one({"filed1": value_to_insert})
client.close()
if __name__ == '__main__':
pool = multiprocessing.Pool(5)
results = [apply_async(pool, write_to_db, args=(x,)) for x in ['one', 'two', 'three']]
output = [p.get() for p in results]
print(output)
produces error:
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Python34\lib\multiprocessing\pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "C:\...\temp2.py", line 10, in run_dill_encoded
return fun(*args)
File "C:\...\temp2.py", line 21, in write_to_db
client = pymongo.MongoClient('localhost', 27017)
NameError: name 'pymongo' is not defined
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:/.../temp2.py", line 32, in <module>
output = [p.get() for p in results]
File "C:/.../temp2.py", line 32, in <listcomp>
output = [p.get() for p in results]
File "C:\Python34\lib\multiprocessing\pool.py", line 599, in get
raise self._value
NameError: name 'pymongo' is not defined
Process finished with exit code 1
What is wrong?

Answer 1

As I mentioned in the comments, you need to put an import pymongo inside the function write_to_db. This is because when the function is serialized, it does not take along any of the global references with it when it is shipped to the other process space.