Hello Prefect Community, Great to be here! I am running a flow on prefect2 with a few tasks which have dependencies on local modules. We are running the tasks on a dask cluster and are using a s3 block storage. The issue I run into is recreating the local dependencies on the dask workers. Any suggestions on how you go about ensuring that the script environment is available to the dask worker?

are your dependencies in the same folder as your flow? Or are these python package dependencies?

They are in the same project folder as the flows and get uploaded to s3 bucket when building the deployment. These dependencies are local scripts, not python package dependencies.

my_flow.py

has tasks which uses functions/classes from

my_script1.py

,

my_script2.py

and so-on.

project_name/
  internal_dependencies
    my_script1.py
    my_script2.py
    ...
   my_flow.py

Interesting. I am doing something similar, and its been working for me

Yes, I'm getting a similar error. How did you resolve this? @Mike Grabbe What's an in-process flow, can you share any documentation/link?

When I deploy the flow, its using the

--infra process

parameter

Isn’t the agent decoupled from the Dask Runner? Ideally the dask runners should be able to load these dependencies and it shouldn’t matter much if the agents are @Mike Grabbe