05/18/2023, 8:06 PM
Hi all, I am attempting to use Prefect as a tool for executing a calendar-day backfill of a sequential series of steps (that is, for each calendar day, run a bunch of steps in series), with a concurrency limit on the number of simultaneous backfill executions. The series of steps fits well into the flow and task model w/ SequentialTaskRunner. Where I am getting hung up a bit is on the backfill mechanism, and also, the parallel execution of flows with the concurrency limit. I've looked at Prefect "deployments", but that seems more suited to some sort of prolonged forward fill. Is there a way to queue up many flow-runs whose parallelism is governed by a work pool? Each time I invoke a flow, it's blocking on execution completion. Apologies in advance if it's blatantly obvious in your docs and I've missed it 🙂