I have a flow which takes approximately 2 days to run, plus/minus several hours each time (this is fine). I want another run of the same flow to begin right away on completion (regardless of success or failure). Only one run of this flow must ever be running.
How might this continuous looping flow situation be achieved?
Asked Marvin but didn’t get a very satisfactory answer.
✅ 1
Simon
07/21/2023, 4:58 PM
Perhaps a custom flow state handler which creates a new flow run instance?
• If you're using PrefectCloud you can do this easily with automations https://docs.prefect.io/2.11.0/cloud/automations/
• If you're hosting prefect yourself, you can simulate what the automation is doing via
run_deployment()
and calling it at the end of flow run (maybe using on_completion/on_failure hooks
either way to ensure only one of the flows is running at a time I would use a dedicated
work_pool
with
concurrency_limit=1
s
Simon
07/21/2023, 5:10 PM
Thanks for the suggestion, much appreciated. I’m self-hosting so will try the run_deployment method at the end of the @flow-decorated function
Simon
07/21/2023, 5:13 PM
It’s a shame an agent cannot grab work from multiple work pools, as I don’t want to dedicate an agent to servicing only this flow
j
Jake Kaplan
07/21/2023, 5:15 PM
you can create a work queue on that pool and set it's concurrency limit to 1
🙌 1
s
Simon
07/21/2023, 5:16 PM
Could tag the first (serial-blocking) task in the flow, and then set task run concurrency of 1 on that tag. A bit hacky as two flow runs would be live, one would be effectively paused. Your way is better 👍
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.