<@ULVA73B9P> today we noticed that a new hourly sc...
# ask-marvin
s
@Marvin today we noticed that a new hourly schedule is not getting any scheduled upcoming runs. Claude thinks it's because of this:
Copy code
Root Cause: Prefect scheduler max_runs limit exceeded

  The Prefect server has these scheduler settings:

  ┌────────────────────────────────────────┬───────┬────────────────────────────────────────────┐
  │                Setting                 │ Value │                  Meaning                   │
  ├────────────────────────────────────────┼───────┼────────────────────────────────────────────┤
  │ max_runs                               │ 100   │ Max total SCHEDULED runs per deployment    │
  ├────────────────────────────────────────┼───────┼────────────────────────────────────────────┤
  │ min_runs                               │ 3     │ Min future runs per schedule               │
  ├────────────────────────────────────────┼───────┼────────────────────────────────────────────┤
  │ deployment_schedule_max_scheduled_runs │ 50    │ Max scheduled runs per individual schedule │
  └────────────────────────────────────────┴───────┴────────────────────────────────────────────┘

  The deployment has 54 active schedules. With min_runs: 3, the scheduler needs at least 54 × 3 = 162 runs — but the max_runs: 100 cap blocks it. The 52 older daily schedules already consume 137 scheduled runs, well over the 100 cap. When the scheduler evaluates the new schedule (65e822ed), the total already exceeds max_runs, so it creates 0 runs for it.

  This also explains why b0aa8933 (the 30-min schedule) has completed runs but 0 future runs — it got into the rotation earlier when there were fewer schedules, but is now being squeezed out of future scheduling.
Can you confirm, and suggest a good solution for this? My concern is any hardcoded limit will become obsolete when there are growing number of schedules being created. Also, when this happens, are we supposed to see any warnings/errors anywhere?
m
thought for 631.0 seconds
Short answer: the “max_runs cap is blocking your new schedule” explanation doesn’t line up with how the Prefect 3.x scheduler actually works. What’s really happening in 3.x - The scheduler evaluates each active schedule on a deployment independently. For each schedule it tries to ensure at least
min_runs
future runs and at least
min_scheduled_time
of coverage, and it will generate up to
max_runs
per schedule. - There is no aggregate “max future runs per deployment” cap that would prevent a new schedule from getting runs just because other schedules have a lot of runs. - The
deployment_schedule_max_scheduled_runs
setting you listed is only used as a validation ceiling for a per-schedule “max_scheduled_runs” field on the schedule schema; it is not used at runtime to throttle scheduling across a deployment (and, today, the per-schedule
max_scheduled_runs
value is not enforced by the scheduler when generating runs). Implication: with 54 schedules, the scheduler will attempt to give each schedule its own
min_runs
buffer. A new hourly schedule should still receive at least
min_runs
future runs, regardless of how many runs other schedules have. Why you might be seeing 0 upcoming runs Common causes I’d check first: - The schedule is paused/inactive. - The schedule has a start time in the future or a timezone/anchor that’s pushing next occurrences out of the window you’re looking at. - The scheduler just hasn’t picked it up yet (the “recent deployments” scheduler loop should pick it up within a few seconds, but logs will confirm). - You’re on an older 3.x patch where the deployment-selection query behaved differently. If so, upgrading to the latest 3.x is recommended. How to verify quickly - Inspect the deployment and its schedules:
Copy code
prefect deployment inspect <deployment-name>
  prefect deployment schedule ls <deployment-name>
- If the schedule shows “paused”, resume it:
Copy code
prefect deployment schedule resume <deployment-name> --schedule-id <SCHEDULE_ID>
- List future flow runs (filter by flow name for clarity):
Copy code
prefect flow-run ls --flow-name <flow-name> --state SCHEDULED --limit 50
- Check server logs for the scheduler. You’ll typically only see lines like “Scheduled N runs.” There are not currently warnings when limits or selection logic prevent scheduling; it’s pretty quiet unless there’s an exception. Recommendations to make this robust as schedules grow - Keep
min_runs
small (e.g., 1–3) and
min_scheduled_time
modest (e.g., 1h–6h) so each schedule gets a thin rolling buffer without creating large backlogs. - If you truly need many independent cadences, consider splitting them across multiple deployments instead of packing dozens of schedules into one deployment. It keeps scheduling work smaller and isolates issues. - If you suspect you’re hitting throughput constraints, you can tune these (self-hosted): -
PREFECT_API_SERVICES_SCHEDULER_DEPLOYMENT_BATCH_SIZE
-
PREFECT_API_SERVICES_SCHEDULER_INSERT_BATCH_SIZE
-
PREFECT_API_SERVICES_SCHEDULER_LOOP_SECONDS
- Upgrade to the latest 3.x if you’re not already on it. To your last question about warnings/errors - There are no explicit warnings when a schedule gets “starved” or when any limits are met; the scheduler logs are minimal. You’ll generally only see a total count of runs scheduled per cycle or an exception if something fails. If you can share: - Your Prefect version - The output of:
Copy code
prefect deployment schedule ls <deployment-name>
  prefect flow-run ls --flow-name <flow-name> --state SCHEDULED --limit 50
- Whether this is Prefect Cloud or self-hosted, and any relevant scheduler log snippets …I can help pinpoint why that specific hourly schedule is at 0 and suggest the precise fix. Helpful docs: - Schedules overview: https://docs.prefect.io/latest/concepts/schedules/ - CLI reference (deployment/schedules): https://docs.prefect.io/latest/cli/
- Settings reference (scheduler): https://docs.prefect.io/latest/reference/settings/
s
@Marvin we noticed that new schedules can sit for hours before getting picked up and scheduled runs according to the set interval. We did an experiment by 1. creating a schedule that runs hourly (we already have 56 other schedules that run on different intervals) 2. wait and confirm no runs are scheduled for at least 2 hours 3. PATCH the deployment and it started to have scheduled runs within a minute: ⏺️ Bash(# Trivial PATCH to bump the deployment's updated timestamp curl -s -X PATCH <prefect_url> ⎿ (No output) ⏺️ Bash(# Verify the updated timestamp changed curl -s <prefect_url> 2>&1 | pytho…) ⎿ Deployment updated: 2026-02-19T024123.474894Z ⏺️ Timestamp bumped to 024123. Now let's wait for the scheduler cycle and check. ⏺️ Bash(sleep 15 && curl -s -X POST <prefect_url> -H 'Content-Type: application/json' -d '{ "flow_runs": {…) ⎿ Total SCHEDULED runs: 60 SCHEDULED runs for 89e33458 (new): 2 2026-02-19T032249.782143Z | auto: True … +4 lines (ctrl+o to expand) ⏺️ That's the confirmation. Within 15 seconds of bumping the deployment's updated timestamp, both schedules got picked up: - 89e33458 (new): 2 scheduled runs at 03:22 and 04:22 - 65e822ed (prev): 2 scheduled runs at 03:10 and 04:10 Root cause confirmed: Adding a schedule via POST /deployments/{id}/schedules does not bump the deployment's updated timestamp. The Prefect scheduler's "recent deployments" loop uses that timestamp to decide what to re-evaluate, so new schedules are invisible until either: - The scheduler's slower full-deployment sweep eventually processes it (took ~5-7 hours last time) - Something else bumps the deployment's updated timestamp Fix options: 1. In our code: After creating a schedule, do a trivial PATCH to the deployment to bump updated 2. Report to Prefect: This is arguably a bug — POST /deployments/{id}/schedules should make it evaluate and schedule runs for it immediately
m
thought for 161.3 seconds
Great sleuthing — I can confirm your root cause analysis is correct for Prefect 3.x. What’s happening - Adding a schedule via
POST /deployments/{id}/schedules
creates
DeploymentSchedule
rows but does not update the parent
Deployment
row, so
deployment.updated
does not change. - The “recent deployments” scheduler loop only re-evaluates deployments whose
updated
timestamp changed recently; your new schedules are invisible to that loop. - The main scheduler sweep only considers deployments that “need” more future runs (based on min_runs/min_window). If a deployment already has enough future runs from existing schedules, it may not be revisited for a while — which matches your 5–7 hour observation. Code references (Prefect 3.x) - Recent-deployments loop filters by `deployment.updated`: scheduler.py - Create-schedules endpoint only inserts schedule rows (doesn’t touch the deployment): api/deployments.py and models/deployments.py -
updated
is only bumped on UPDATE of the deployment row: orm_models.py - Scheduler settings (loop intervals, min runs/window): settings/models/server/services.py Why your experiment worked - Your trivial PATCH produced an UPDATE on the deployment row, bumping
deployment.updated
. The recent-deployments loop then picked it up within seconds and generated runs for the new schedules. Workarounds you can use today - After creating schedules, issue a no-op PATCH to the deployment to bump `updated`:
Copy code
curl -s -X PATCH "<prefect_url>/api/deployments/<deployment_id>" \
       -H "Content-Type: application/json" \
       -d '{"description": "touch to bump updated"}'
Any PATCH that results in an UPDATE to the deployment row is sufficient. Your approach is spot on. - Alternatively, if you’re updating deployments via code, ensure the schedule change happens as part of a deployment update (e.g., re-deploy), which will naturally bump
updated
. Recommendation to Prefect (bug/UX fix) -
POST /deployments/{id}/schedules
should also “touch” the deployment (or emit an event) so the recent-deployments loop re-evaluates it immediately. - Another robust option would be for the recent-deployments query to consider recent changes to schedules for a deployment, not just the deployment’s
updated
. Next steps - I’ll file this as a bug with our team. If you can share: - Your Prefect version (and whether this is Cloud or self-hosted) - Any non-default scheduler settings (loop intervals, min_runs, min_window) - A minimal repro (the steps you posted are already excellent) …I’ll include them to speed up triage. In the meantime, your PATCH-after-POST workaround is the right mitigation to ensure new schedules are picked up immediately.
s
@Marvin Prefect 3.6.16 . No non-default settings
m
thought for 164.2 seconds
Thanks — noted: Prefect 3.6.16 with default scheduler settings. Your diagnosis still applies in 3.6.16:
POST /deployments/{id}/schedules
doesn’t update the parent
Deployment
, so
deployment.updated
remains unchanged and the recent-deployments loop won’t re-evaluate it. Only the slower/full sweep will eventually pick it up unless you touch the deployment. Key references - Recent-deployments loop filters by `deployment.updated`: server/services/scheduler.py - Create-schedules endpoint only inserts schedule rows: server/api/deployments.py and server/models/deployments.py -
updated
is only bumped on UPDATE of the deployment row: server/database/orm_models.py Workaround (Python client)
Copy code
from uuid import UUID
from prefect import get_client
from prefect.client.schemas.actions import DeploymentUpdate, DeploymentScheduleCreate
from prefect.client.schemas.schedules import IntervalSchedule

deployment_id = UUID("YOUR-DEPLOYMENT-ID")

async def add_schedule_and_bump():
    async with get_client() as client:
        # 1) Add the schedule
        await client.create_deployment_schedules(
            deployment_id=deployment_id,
            schedules=[
                DeploymentScheduleCreate(
                    schedule=IntervalSchedule(interval=3600),  # hourly
                    active=True,
                    slug="hourly-auto"
                )
            ],
        )

        # 2) Trivial update to bump `deployment.updated`
        await client.update_deployment(
            deployment_id=deployment_id,
            deployment=DeploymentUpdate(description="touch to bump updated")
        )
Workaround (HTTP)
Copy code
curl -s -X PATCH "<prefect_api>/deployments/<deployment_id>" \
  -H "Content-Type: application/json" \
  -d '{"description": "touch to bump updated"}'
Next steps - I’ll file this as a bug so
POST /deployments/{id}/schedules
also “touches” the deployment (or the scheduler considers recent schedule changes). I’ll share the link here once it’s created. - Quick confirm: are you on Prefect Cloud or self-hosted (OSS)? This helps us route the fix and validate in the right environment.
s
@Marvin we are on self-hosted
m
thought for 74.3 seconds
Got it — thanks for confirming you’re self-hosted. Given you’re on Prefect 3.6.16 with default settings, your observations line up with the current scheduler behavior: - Recent-deployments loop (default every 5s) only looks at deployments with a recently-updated
deployment.updated
timestamp. - Adding schedules via
POST /deployments/{id}/schedules
doesn’t modify that timestamp, so the new schedules won’t be picked up until either a deployment UPDATE occurs or the main/full sweep decides the deployment “needs” more runs. If it helps operationally, here are the relevant scheduler defaults and knobs in self-hosted: - Main loop interval: 60s (env:
PREFECT_SERVER_SERVICES_SCHEDULER_LOOP_SECONDS
) - Recent deployments loop: 5s (config-only; no env var alias) - Min future runs: 3 (env:
PREFECT_SERVER_SERVICES_SCHEDULER_MIN_RUNS
) - Min scheduled time window: 1 hour (env:
PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME
) Links: settings/models/server/services.py and server/services/scheduler.py Short-term mitigation - Keep doing your no-op PATCH after creating schedules (either via HTTP or Python client’s
update_deployment(DeploymentUpdate(...))
) to bump
deployment.updated
. I’ll raise this upstream as a bug/UX fix for self-hosted OSS so that adding schedules touches the deployment (or the recent-deployments loop considers recent schedule changes). If you’re okay with it, I can include your repro notes (very helpful) when I file it.