I am trying to run prefect flow on Dask cluster using kubernetes and AWS. I was able to import "prefect" and also print its version. But when I execute the flow, it says

ModuleNotFoundError: No module named 'prefect'

. I have installed all the dependencies that need to be installed by Dask-worker and Dask-jupyter using yaml file.

You need to also install

prefect

on the dask workers and the scheduler

@Chris White I actually did so using this .yaml file

Interesting; did you also install

prefect

on the scheduler?

I haven't specified that. Above snippet is all how my yaml looks like

Sorry, I think I’m missing something; your workers must be connecting to a dask scheduler somewhere, right?

So dask cluster initiates 2 dask workers and a scheduler, when run on kubernetes, as I am using 2 instances. So I have also setup Jupyter Hub using Helm. I am executing my entire flow in the Jupyter Hub.

so where do you see the

ModuleNotFoundError

, and could you describe how you are submitting your Flow to run on the dask cluster?

Untitled

thank you - and where do you call this script from?

from Jupyter Hub and this hub is based on kubernetes and Dask cluster

I assume Jupyter Hub uses conda environments; have you verified that you are running this script in the correct environment?

No I haven't. I think the error is because I haven't installed prefect on scheduler. Will try working on that

ok let me know! It would be really nice to have a working template for how people can work with Prefect on Jupyter Hub that we post on GitHub

Sure Chris. Thanks so much for your help

anytime @Aakarsh Nadella!

@Aakarsh Nadella FWIW, we had the Prefect Advanced Features Tutorial working from a notebook using the DaskExecutor on our Dask cluster running on AWS EKS with dask-kubernetes. Here's the exact notebook that we used and I'm happy to try to answer any questions.