https://prefect.io logo
j

Joe Goldbeck

07/11/2022, 7:15 PM
Hi all: Using Prefect 1.x, is it possible to have timeout/expiration for late flow runs? We run an export job every 15 minutes, so if our agent goes down, these can accumulate rather quickly. When the agent comes back up, we do not want to go back and run all the missed scheduled runs, we just want to pick back up from wherever it is.
k

Kevin Kho

07/11/2022, 7:16 PM
Yes see Automations. Specifically you can cancel a flow run if it is late by a certain amount of time using the Flow SLA
j

Joe Goldbeck

07/11/2022, 7:18 PM
Thanks! Is this kind of ability Cloud only? e.g. if we are running our own Server, we could not clear these out except by hand?
k

Kevin Kho

07/11/2022, 7:19 PM
Yes because it relies on services not shipped with Server. For Server, your best bet is implementing something like this to cancel all of the extra runs
j

Joe Goldbeck

07/11/2022, 7:22 PM
I see ty! I am surprised there is no simple way to do this, but I should have done better research, hah. One my quick q if you have the time
in Prefect 2, I see
timeout_seconds
in Flow. Would that expire a flow if it hasn’t been started running by timeout_seconds after the scheduled start time? https://orion-docs.prefect.io/api-ref/prefect/flows/
k

Kevin Kho

07/11/2022, 7:23 PM
Yes that’s right
j

Joe Goldbeck

07/11/2022, 7:24 PM
Ok that’s more sane 🙂. Sounds like a reason to update to 2.0 sooner rather than later. Ty so much for the quick replies!
k

Kevin Kho

07/11/2022, 7:25 PM
Of course!
z

Zanie

07/11/2022, 7:41 PM
Ah actually that’s not how
timeout_seconds
works — that’s a timeout on flow run time
There is no 2.x feature for “flow scheduled start time” enforcement yet
🙁 1
It should be trivial to write a service though if you’re not trying to have it as a configurable feature per-flow
j

Joe Goldbeck

07/11/2022, 7:48 PM
🤔 What do you mean by “write a service”?
Like build something else outside of Prefect that polls the database and cleans up late runs, or…?
z

Zanie

07/11/2022, 7:50 PM
For v1 server I’d add an image to the docker-compose that queries for late runs in batches and updates their state, yeah
You could also subclass the agent to check differences between scheduled start time and the current time
🤔 1
j

Joe Goldbeck

07/11/2022, 7:54 PM
Subclass the agent sounds like getting into internals?
For the docker-compose, that sorta bumps the problem along a bit since then something needs to run and monitor those jobs 😛. Tbh I’m a bit surprised this isn’t built-in to the scheduler – should have read the docs more carefully. Appreciate the help!
3 Views