m

    Max

    1 year ago
    Hi everyone! Does dask executor support versioning? Let's say I have two versions of a codebase for my tasks, each lives in a separate docker container. I'd like to be able to run both versions, e.g. flow1 runs inside container1, and flow2,3 run inside container2. This could be achieved without dask AFAIK, but dask is really handy for resources management, such as GPU memory.
    Chris White

    Chris White

    1 year ago
    Hi Max! This depends largely on how you are leveraging dask. If you are using an ephemeral or a local dask cluster, then it will work off the shelf. If you are managing an external dask cluster, you’ll need to make sure all versions match (e.g., dask, cloudpickle, prefect, etc.)