How to run parallelized python jobs on yarn using Dask? - hadoop

Question

I have a couple of questions on using Dask with Hadoop/Yarn.
1 ) How do I connect Dask to Hadoop/YARN and parallelize a job?
When I try using:
from dask.distributed import Client
client = Client('Mynamenode:50070')
It results in the error:
CommClosedError: in : Stream is closed: while trying to call remote method 'identity'
Should I pass the address of the name node or a datanode? Can I refer Zookeeper instead?
2 ) How do I read data from HDFS, using Dask and HDFS3?
When I try to read a file using:
import dask.dataframe as dd
import distributed.hdfs
df = dd.read_csv('hdfs:///user/uname/dataset/temps.csv')
It results in the following error :
ImportError: No module named lib
I have tried uninstalling and reinstalling hdfs3, but the error is still persistent.
I have installed knit and tried launching yarn containers using this example:
http://knit.readthedocs.io/en/latest/examples.html#ipython-parallel
This fails with a security error.
I do not have sudo access on the cluster, so installing any packages on each node in the cluster is out of the question, the only installations I can do are via conda and pip under my userid.
Finally, it would be greatly helpful, if someone could post a working example of Dask on Yarn.
Greatly appreciate any help,

Answer 1

The simplest implementation of dask-on-yarn would look like the following
install knit with conda install knit -c conda-forge (soon the package "dask-yarn" will be available, perhaps a more obvious name)
The simplest example of how to create a dask cluster can be found in the documentation. Here you create a local conda environment, upload it to HDFS and have YARN distribute it to workers, so you do not need sudo access.
Note that there are a lot of parameters that you can pass, so you are encouraged to read the usage and troubleshooting parts of the docs.
Specific answers to questions
1) Client('Mynamenode:50070') - hadoop does not know anything about dask, there is no reason that a namenode server should know what to do with a dask client connection
2) No module named lib - this is very strange and perhaps aa bug that should be logged by itself. I would encourage you to check that you have compatible versions of hdfs3 (ideally the latest) in the client and any workers
3) fails with a security error - this is rather nebulous and I cannot say more without further information. What security tdo you have enabled, what error do you see? It may be that you need to authenticate with kerberos but have not run kinit.