Hi guys (and girls), Currently we are using Prefect server with 1 agent and a default Dask executor with Dask scheduler and workers. We are running into problems with using local imports and then registering these imports in our flow. We get a "ModuleNotFoundError" for the given module, for example:

from test.package1 import test1

Would result into "ModuleNotFoundError: No module named 'test'". We are not using Docker images thus on registering we want this to be resolved instead of installing our package somewhere, which we constantly need to update. The same issue happens here https://github.com/PrefectHQ/prefect/issues/2797 except here they are using Docker images, while we are not.

Hey @Fabrice Toussaint, a bit unclear here. If you don’t use Docker, how will you get this

test

library to the Dask scheduler and workers?

That is exactly the question, isn't that possible to do that on registration? Instead of preregistering every utils packages needed for a flow to register.

Ah no this is not something that Prefect registration does or can do. Prefect just keeps track of the metadata of the Flow (storage, runconfig), but doesn’t serialize other packages (limitation of

cloudpickle

). If you’re expecting these modules to be installed on the cluster, you can defer these imports inside of tasks so you don’t need them on registration side. But when it comes to the Dask cluster, you pretty much need an image. Are you spinning up a cluster for your flow or using an existing one? You normally have to provide an image to spin up the workers. If you are using an existing one, it’s a bit trickier to get these custom modules installed but it can be done. Is your issue with Docker the long build time for the Docker image? Cuz I do have a tutorial that may help (but I suspect you might know how to do this)