https://prefect.io logo
s

Simon

07/21/2023, 4:49 PM
I have a flow which takes approximately 2 days to run, plus/minus several hours each time (this is fine). I want another run of the same flow to begin right away on completion (regardless of success or failure). Only one run of this flow must ever be running. How might this continuous looping flow situation be achieved? Asked Marvin but didn’t get a very satisfactory answer.
1
Perhaps a custom flow state handler which creates a new flow run instance?
j

Jake Kaplan

07/21/2023, 5:08 PM
• If you're using PrefectCloud you can do this easily with automations https://docs.prefect.io/2.11.0/cloud/automations/ • If you're hosting prefect yourself, you can simulate what the automation is doing via
run_deployment()
and calling it at the end of flow run (maybe using on_completion/on_failure hooks either way to ensure only one of the flows is running at a time I would use a dedicated
work_pool
with
concurrency_limit=1
s

Simon

07/21/2023, 5:10 PM
Thanks for the suggestion, much appreciated. I’m self-hosting so will try the run_deployment method at the end of the @flow-decorated function
It’s a shame an agent cannot grab work from multiple work pools, as I don’t want to dedicate an agent to servicing only this flow
j

Jake Kaplan

07/21/2023, 5:15 PM
you can create a work queue on that pool and set it's concurrency limit to 1
🙌 1
s

Simon

07/21/2023, 5:16 PM
Could tag the first (serial-blocking) task in the flow, and then set task run concurrency of 1 on that tag. A bit hacky as two flow runs would be live, one would be effectively paused. Your way is better 👍