Hi all! I'm getting start with Prefect. It's very pleasing.
I have a question about runs of the same flow overlapping in time. For example if there are several scheduled flow runs but no agent running, they might start to stack up. If then an agent is started then all the flow runs (for this single flow) will start at the same time. A similar scenario could also come about because a flow takes a long time to run and another starts before it's finished.
Obviously I'm writing my flow so that runs don't interfere with other runs, but nevertheless I was wondering if there is a way to make this not happen. I.e. a flow run checks to see if there are other flow-runs currently running, and skips itself if that's the case. I can't see that I can do that using State, or by using a different Schedule. Is there an idiomatic way to do this?
10/28/2020, 1:27 PM
Hi @James Cole in order to off-the-shelf prevent runs from starting at the same time it would require a system for locking the concurrency of the flows. Right now there are flow level concurrency limits in the Prefect Cloud offering and considerations to include them in Prefect server (https://github.com/PrefectHQ/server/issues/81)
There is also a more granular concurrency limiting feature in Prefect Cloud that allows you to limit the global concurrency of certain tasks based on their tags, but access to this feature requires an appropriate Cloud license.
10/28/2020, 1:55 PM
Hi @josh. Thanks for those details! I'll keep an eye on that issue.