Is it possible to set priorities for flows with the kubernetes agent? Ideally I'd like to have high priority flows be able to pre-empt and pause lower priority flows if the cluster is maxed out
k
Kevin Kho
09/25/2021, 1:44 AM
Hey @Aaron Ash, we don’t have a concept of priority (thus far) because this tends to be a task queue concept where you have a bunch of similar tasks and workers pick them up upon availability. Prefect is more concerned with batch jobs rather than processing from a queue.
I think the analogous thing here is having priorities for tasks to be picked up by a Dask Executor. This has come up in our discussions and is on our radar. Dask has support this as well, but we don’t have any immediate roadmap for it.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.