Clemens

    Clemens

    1 year ago
    Hi everyone, I am trying to connect prefect to a local dask cluster to test out distributed calculation of a flow that I want to register to a dask cluster in the cloud later. I followed the tutorial https://docs.prefect.io/core/advanced_tutorials/dask-cluster.html and that works great as long as everything is defined in one python file. But if I want to import any function from a second file running it results in a
    ModuleNotFoundError
    . Does anybody have any experience on how Dask is handling these imports?
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi! This kind of depends how you’re storing your flows. Perhaps this recent thread would be helpful! https://prefect-community.slack.com/archives/CL09KU1K7/p1603404858002800?thread_ts=1603396451.488900&cid=CL09KU1K7
    Clemens

    Clemens

    1 year ago
    thanks, i will have a look! to answer the question: I haven’t set up any storage. Just running the tutorial file as is and moving one function to another file and then reimporting breaks it.
    nicholas

    nicholas

    1 year ago
    That makes sense @Clemens - Prefect doesn't do anything special to package your flow and so relies on imports existing in the runtime environment; dependencies should either be pip-installable or present in the environment. As @Michael Adkins mentioned, this is usually resolved by packaging your flows using whatever storage is appropriate for your setup.
    Michael Adkins

    Michael Adkins

    1 year ago
    If storage is not set, it defaults to using LocalStorage as described at https://docs.prefect.io/orchestration/execution/storage_options.html#local
    There’s also a stack overflow answer on this https://stackoverflow.com/questions/63881231/prefect-modulenotfounderror-when-running-from-ui — generally the trick is ensuring that your files / module are on the python path while being executed so they can be imported
    Clemens

    Clemens

    1 year ago
    got it running, thank’s a lot!!