Hello! I have set up a notification automation to notify on late flow runs, and I have set the

PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS

on the worker to be 30 minutes (basically I just need the notification if it's REALLY late), however, we are getting notified after a couple of seconds of the run attempting to start. What am I missing?

hi @Gemma - have you configured this setting on the server or just on the worker?

PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS

is a server-side configuration (since it affects the loop service that marks flow runs as late), so you'd need to set it there (it won't have an effect if you just set it where the worker runs)

We are using Prefect Cloud with an EC2 instance running docker as the worker. I haven't been able to find any docs about this really so sorry for the potentially stupid question, but where would I need to set it? In the docker container this is the script we run:

prefect profile create worker
prefect profile use worker
...
prefect config set PREFECT_API_SERVICES_LATE_RUNS_AFTER_SECONDS=1800
...

WORK_POOL_NAME=${ENV}-work-pool
prefect work-pool create --type process "$WORK_POOL_NAME" || true
prefect work-pool set-concurrency-limit "$WORK_POOL_NAME" 5
prefect worker start --pool "$WORK_POOL_NAME"

no worries! where are you running prefect server?

I may be doing the wrong thing, as I say I've not been able to find docs for this. In Prefect Cloud, I have configured an automation to notify us via Slack on late flow runs. However, how do I configure (within Prefect Cloud) what gets flagged as a "late" run?

how do I configure (within Prefect Cloud) what gets flagged as a “late” run?

short answer is that you do not. We run a service in Prefect Cloud that’s responsible for marking flow runs as late (that’s what I meant earlier by mentioning that its a server-side setting) if you just need to be notified only after flow runs have been stuck in Late for 30 minutes, you could just change your automation from looking at a flow run entering

Late

to looking for flow runs that stay in

Late

for 30 minutes with something like this

Ahh perfect - thanks! As some feedback, it would be great if there was some documentation around this, as it took a lot of research to try and work out how to do it, and (as you can see from my obvious blunder), what I could find was a bit misleading for me. I really appreciate you taking the time to provide the answer though 😄

thanks for the feedback! we'll be working a lot on the docs in the near future so its especially useful