<@ULVA73B9P> What are the potential issues with scheduling a flow like this with Cron in my prefect....
t
@Marvin What are the potential issues with scheduling a flow like this with Cron in my prefect.yaml file? I'm seeing some failures of my flow runs right at the 9:00am and 3:00pm times every day due to some starting late and overlapping with one another. I am using Prefect 2.0.0
Copy code
schedules:
  - cron: "30-59 8 * * 1-5"
    timezone: America/Chicago
  - cron: "* 9-14 * * 1-5"
    timezone: America/Chicago
  - cron: "0-30 15 * * 1-5"
    timezone: America/Chicago
m
thought for 794.8 seconds
Thanks for the details — a few things to watch out for with that schedule, especially on Prefect 2.0.0: What’s likely happening - You’re scheduling every minute across long windows. If a run takes >60s (even occasionally), the next minute’s run will start while the previous is still active. The transition minutes (9:00 and 15:00) are where a new cron clock picks up — so any run that is still going from 8:59 or 14:59 can overlap with the first run of the next window and cause resource contention/failures. - In Prefect 2.0.0 there’s no per-deployment “only run one at a time” control. Without a concurrency guard, multiple flow runs will overlap if they’re scheduled faster than they complete. - Multiple cron clocks on one deployment are fine, but they are independent. If a run starts late at the end of one window, it can collide with the first minute of the next window. Ways to prevent overlap in Prefect 2.0.0 1) Put the deployment on its own work queue and set the queue’s concurrency to 1 - This ensures only one flow run is picked up at a time. - Create a queue with a concurrency limit:
Copy code
prefect work-queue create my-queue --limit 1
- Or set a limit on an existing queue:
Copy code
prefect work-queue set-concurrency-limit <queue-name-or-id> 1
- Assign your deployment to that queue (via the deployment’s work queue setting or in the UI). 2) Use a task-level concurrency “mutex” with tags - Concurrency limits in 2.x are task-level. Wrap the body of your flow in a single task tagged with a shared lock and set that tag’s limit to 1. Even if two flow runs start, one will wait at the critical section.
Copy code
from prefect import flow, task

  @task(tags=["my-lock"])
  def do_work():
      # your critical work here

  @flow
  def my_flow():
      do_work()
Then create the limit:
Copy code
prefect concurrency-limit create my-lock 1
- Note: This doesn’t stop the second flow run from starting, but it will block it at the task boundary so your critical work doesn’t overlap. - Docs: Global concurrency limits (2.x) 3) Adjust schedule or runtime expectations - If possible, schedule less frequently (e.g., every 2–5 minutes) or ensure the flow completes in <60s consistently. - You could also nudge the boundary minutes (e.g., start the afternoon window at 15:01) to avoid transitions, if that fits your needs. Other considerations - Timezone and DST: You’re setting America/Chicago, which is good. DST transitions won’t affect 900/1500 most days, but be mindful around clock changes. - Prefect 2.0.0 is quite old — the scheduler and work-queue behavior improved substantially in later 2.x. If you can upgrade within 2.x (or to 3.x), you’ll benefit from more reliable scheduling and better worker/queue controls. Relevant docs - Schedules (2.x) - Global concurrency limits (2.x) If you can share: - Rough average/95th percentile runtime of the flow - Whether this deployment is alone on its work queue - How many workers you have polling that queue …I can recommend a precise setup (queue limits vs. task lock vs. schedule tweak).