Hey! I've task that is scheduled to run every 4 hours. However sometimes the associated agent isn't running for let's say a day. In that case the pending runs obviously pile up. I'd like to avoid that, especially because there is no point running that many runs the next day. I can imagine 3 ways to implement such behavour, unfortunately I didn't find anything in the docs so far:
1. set a limit of pending runs of a given fllow
2. timeout a pending run after some time
3. add a task to the flow that checks the scheduled time and skips the actual task if above some threshold
I assume 1 and 2 are not available. 3 should be possible somehow. Any pointers on how to implement this?
✅ 1
c
Chris White
04/06/2021, 9:39 AM
Hi Wolfgang — #2 will be available as a feature on the standard tier in the next week or two. In the meantime, you can implement #3 by using
prefect.context["scheduled_start_time"]
to perform your task check, which is a datetime object representing the scheduled start time for the flow run
🙌 1
s
Samuel Hinton
04/06/2021, 9:43 AM
We have a similar issue and followed the advice Chris laid out to fix it. Our initial task in the flow has a max_late timedelta set, and within the task we run this:
Copy code
from prefect.engine.signals import SKIP
if prefect.context["scheduled_start_time"] + max_late < datetime.datetime.now(tz=pytz.utc):
raise SKIP("Task is too late, skipping")
upvote 2
w
Wolfgang Steitz
04/06/2021, 9:50 AM
Cool! Thank you guys. I'll try Samuel's code snippet for now and looking forward to have "run timeouts" available soon
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.