Hi everyone! Does dask executor support versioning? Let's say I have two versions of a codebase for my tasks, each lives in a separate docker container. I'd like to be able to run both versions, e.g. flow1 runs inside container1, and flow2,3 run inside container2. This could be achieved without dask AFAIK, but dask is really handy for resources management, such as GPU memory.
1 year ago
Hi Max! This depends largely on how you are leveraging dask. If you are using an ephemeral or a local dask cluster, then it will work off the shelf. If you are managing an external dask cluster, you’ll need to make sure all versions match (e.g., dask, cloudpickle, prefect, etc.)