Hi everyone! Does dask executor support versioning? Let's say I have two versions of a codebase for my tasks, each lives in a separate docker container. I'd like to be able to run both versions, e.g. flow1 runs inside container1, and flow2,3 run inside container2. This could be achieved without dask AFAIK, but dask is really handy for resources management, such as GPU memory.
c
Chris White
11/03/2020, 3:52 PM
Hi Max! This depends largely on how you are leveraging dask. If you are using an ephemeral or a local dask cluster, then it will work off the shelf. If you are managing an external dask cluster, you’ll need to make sure all versions match (e.g., dask, cloudpickle, prefect, etc.)
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.