I see in Dask that there's a priority parameter for "when there's more work than workers [and] Dask has to decide which tasks to prioritize over others." Is this exposed to Prefect? More specifically, let's say I have a collection of flows and my system is running low on resources, is there a way to tell Prefect to prioritize completing one flow over another? Thanks in advance!
1 year ago
Hi @Marcus Hughes - this isn't possible in Prefect at the moment but if it's something you'd like to see it's worth opening a discussion in Github🙂
1 year ago
So there is this issue for task priority you can chime in. For the Flow level though, I think this is tricky because having a priority is a task queue kind of thing where you have a set of tasks that feed into a set of workers, and then you can set a “priority queue”On the flow level, Prefect isn’t meant to be a task queue (or flow queue) so it will just attempt to start whatever. I know it’s not quite what you want, but have you seen flow concurrency limits?