I am trying to run prefect flow on Dask cluster us...
# prefect-community
a
I am trying to run prefect flow on Dask cluster using kubernetes and AWS. I was able to import "prefect" and also print its version. But when I execute the flow, it says
ModuleNotFoundError: No module named 'prefect'
. I have installed all the dependencies that need to be installed by Dask-worker and Dask-jupyter using yaml file.
c
You need to also install
prefect
on the dask workers and the scheduler
a
@Chris White I actually did so using this .yaml file
c
Interesting; did you also install
prefect
on the scheduler?
a
I haven't specified that. Above snippet is all how my yaml looks like
c
Sorry, I think I’m missing something; your workers must be connecting to a dask scheduler somewhere, right?
where / how are you running this flow?
a
So dask cluster initiates 2 dask workers and a scheduler, when run on kubernetes, as I am using 2 instances. So I have also setup Jupyter Hub using Helm. I am executing my entire flow in the Jupyter Hub.
When flow is executed, it should automatically assign tasks to Dask workers.
c
so where do you see the
ModuleNotFoundError
, and could you describe how you are submitting your Flow to run on the dask cluster?
a
Untitled
c
thank you - and where do you call this script from?
a
from Jupyter Hub and this hub is based on kubernetes and Dask cluster
c
I assume Jupyter Hub uses conda environments; have you verified that you are running this script in the correct environment?
a
No I haven't. I think the error is because I haven't installed prefect on scheduler. Will try working on that
c
ok let me know! It would be really nice to have a working template for how people can work with Prefect on Jupyter Hub that we post on GitHub
a
Sure Chris. Thanks so much for your help
c
anytime @Aakarsh Nadella!
j
@Aakarsh Nadella FWIW, we had the Prefect Advanced Features Tutorial working from a notebook using the DaskExecutor on our Dask cluster running on AWS EKS with dask-kubernetes. Here's the exact notebook that we used and I'm happy to try to answer any questions.
💯 1
I also noticed in your .yaml file above you have
prefect
listed for both
EXTRA_CONDA_PACKAGES
and
EXTRA_PIP_PACKAGES
-- you can just put it in the CONDA one. (If a package is available on conda-forge, put it in the CONDA one. If a package is not, put it in the PIP one.)
👍 1