https://prefect.io logo
Title
a

Aakarsh Nadella

09/16/2019, 9:13 PM
I am trying to run prefect flow on Dask cluster using kubernetes and AWS. I was able to import "prefect" and also print its version. But when I execute the flow, it says
ModuleNotFoundError: No module named 'prefect'
. I have installed all the dependencies that need to be installed by Dask-worker and Dask-jupyter using yaml file.
c

Chris White

09/16/2019, 9:15 PM
You need to also install
prefect
on the dask workers and the scheduler
a

Aakarsh Nadella

09/16/2019, 9:17 PM
@Chris White I actually did so using this .yaml file
c

Chris White

09/16/2019, 9:19 PM
Interesting; did you also install
prefect
on the scheduler?
a

Aakarsh Nadella

09/16/2019, 9:19 PM
I haven't specified that. Above snippet is all how my yaml looks like
c

Chris White

09/16/2019, 9:21 PM
Sorry, I think I’m missing something; your workers must be connecting to a dask scheduler somewhere, right?
where / how are you running this flow?
a

Aakarsh Nadella

09/16/2019, 9:27 PM
So dask cluster initiates 2 dask workers and a scheduler, when run on kubernetes, as I am using 2 instances. So I have also setup Jupyter Hub using Helm. I am executing my entire flow in the Jupyter Hub.
When flow is executed, it should automatically assign tasks to Dask workers.
c

Chris White

09/16/2019, 9:42 PM
so where do you see the
ModuleNotFoundError
, and could you describe how you are submitting your Flow to run on the dask cluster?
a

Aakarsh Nadella

09/16/2019, 9:54 PM
Untitled
c

Chris White

09/16/2019, 9:56 PM
thank you - and where do you call this script from?
a

Aakarsh Nadella

09/16/2019, 9:56 PM
from Jupyter Hub and this hub is based on kubernetes and Dask cluster
c

Chris White

09/16/2019, 9:57 PM
I assume Jupyter Hub uses conda environments; have you verified that you are running this script in the correct environment?
a

Aakarsh Nadella

09/16/2019, 9:58 PM
No I haven't. I think the error is because I haven't installed prefect on scheduler. Will try working on that
c

Chris White

09/16/2019, 9:58 PM
ok let me know! It would be really nice to have a working template for how people can work with Prefect on Jupyter Hub that we post on GitHub
a

Aakarsh Nadella

09/16/2019, 9:59 PM
Sure Chris. Thanks so much for your help
c

Chris White

09/16/2019, 9:59 PM
anytime @Aakarsh Nadella!
j

Joe Schmid

09/17/2019, 12:09 AM
@Aakarsh Nadella FWIW, we had the Prefect Advanced Features Tutorial working from a notebook using the DaskExecutor on our Dask cluster running on AWS EKS with dask-kubernetes. Here's the exact notebook that we used and I'm happy to try to answer any questions.
💯 1
I also noticed in your .yaml file above you have
prefect
listed for both
EXTRA_CONDA_PACKAGES
and
EXTRA_PIP_PACKAGES
-- you can just put it in the CONDA one. (If a package is available on conda-forge, put it in the CONDA one. If a package is not, put it in the PIP one.)
👍 1