<@ULVA73B9P> i updated to 3.6.16 last night, and t...
# ask-marvin
k
@Marvin i updated to 3.6.16 last night, and today we observed a bug where deployments that have 3 schedules stopped producing scheduled runs after a certain point. all other deployments with 1-2 schedules were not affected. toggling the schedules off and on seemed to fix it, i.e. prefect server generated the correct upcoming runs. is this a known bug?
m
thought for 213.1 seconds
n
i (and marvin im sure) would be interested to know the value of your schedule
Copy code
» uvx prefect deployment schedule ls solux-update/circadian-lights -o json | jq
[
  {
    "id": "567baee9-f1f1-47fa-835e-cd1d03f5a0a4",
    "created": "2026-01-11T13:18:05.830849Z",
    "updated": "2026-02-16T18:35:01.742773Z",
    "deployment_id": "a55ef035-4c74-4667-89a9-d730e5c10f3e",
    "schedule": "interval: 0:15:00s",
    "active": true,
    "max_scheduled_runs": null,
    "parameters": {},
    "slug": null
  }
]
m
Thanks for flagging this — and sorry for the disruption. Short answer: I can’t find a confirmed, open bug specific to Prefect 3.6.16 where deployments with exactly three schedules stop producing upcoming runs. Toggling schedules to “reset” upcoming runs does line up with how the scheduler recomputes state, so your workaround makes sense, but I don’t see a 3.6.16-specific report that matches this pattern. What I did find that’s related: - There’s prior discussion about multiple schedules not producing upcoming runs when they share the same cron expression (seen on earlier 3.x) — symptoms look similar: Multiple schedules with the same cron expression are not properly executed (reported on 3.2.2 in Cloud). - Recent releases added schedule slug renaming (“replaces”); if you’ve renamed schedules recently, that’s an adjacent area where we’ve been hardening behavior. See the 3.6.17 notes “Add replaces field for renaming schedule slugs” PR #20460 and the 3.6.16→3.6.17 diff here: Full changelog. A few quick questions to help us narrow this down: - Are you on Prefect Cloud or self-hosting Prefect Server? - Do the three schedules have identical or overlapping cron/intervals (same minute) and the same timezone? - Did you recently rename any schedules (or change schedule names in the UI/API) before this started? - Roughly when did “upcoming runs” stop advancing (timestamp and timezone), and did it affect all three schedules or just one? - After toggling off/on, did all three schedules begin producing runs again, or only some? Suggested next steps and short-term mitigations: - If two or more schedules are meant to fire at exactly the same minute, try giving each schedule a distinct name (slugs derive from names) and see if the issue reappears. If feasible, staggering their minutes by 1–2 minutes can avoid any edge-case de-duplication. - If you recently renamed schedules, try avoiding chained renames for now and keep slugs/names unique. - If you’re self-hosting, grab server logs around the time upcoming runs stopped; that will help us confirm whether the scheduler stopped advancing the cursor for one of the schedules. If you can share a minimal repro (deployment name, the three schedule definitions with cron/interval + timezone, and whether any were renamed), I’m happy to open a GitHub issue on your behalf so the team can dig in. Otherwise, feel free to open one directly and we’ll jump on it.
k
@Nate @Marvin here is the dump. it doesnt seem to show the timezone though. here is the definition in my prefect.yaml prefect.yaml
Copy code
schedules:
      - cron: "0 10 * * 1-5"
        timezone: "UTC"
      - cron: "0 7-23 * * 1-5"
        timezone: "America/New_York"
      - cron: "0 */6 * * 0,6"
        timezone: "America/New_York"
schedule json
Copy code
[
  {
    "id": "8f70ba01-f4d8-4496-890c-03753dd40255",
    "created": "2026-02-11T22:02:55.786018Z",
    "updated": "2026-02-16T19:21:35.572219Z",
    "deployment_id": "0b3db108-7608-413e-9aaf-004ed05e7f3a",
    "schedule": "cron: 0 7-23 * * 1-5",
    "active": true,
    "max_scheduled_runs": null,
    "parameters": {},
    "slug": null
  },
  {
    "id": "bc0a6eee-efa1-47ca-8342-e8e4ecc92903",
    "created": "2026-02-11T22:02:55.786026Z",
    "updated": "2026-02-11T22:02:55.786028Z",
    "deployment_id": "0b3db108-7608-413e-9aaf-004ed05e7f3a",
    "schedule": "cron: 0 */6 * * 0,6",
    "active": true,
    "max_scheduled_runs": null,
    "parameters": {},
    "slug": null
  },
  {
    "id": "626bb6ef-745c-4c87-b9d4-661cd20762cb",
    "created": "2026-02-11T22:02:55.786003Z",
    "updated": "2026-02-11T22:02:55.786011Z",
    "deployment_id": "0b3db108-7608-413e-9aaf-004ed05e7f3a",
    "schedule": "cron: 0 10 * * 1-5",
    "active": true,
    "max_scheduled_runs": null,
    "parameters": {},
    "slug": null
  }
]
we were previously running 3.4.24 for many months without issue.
m
thought for 439.5 seconds
Thanks for sharing the dump — that helps. A couple quick confirmations based on what you posted: - The CLI/json output you pasted is expected to omit timezone. The current
prefect deployment schedule ls -o json
replaces the full schedule object with a human-readable string like
cron: 0 7-23 * * 1-5
, so the timezone isn’t shown there. Your prefect.yaml shows the correct timezones, and the API does store them. - Your three schedules aren’t identical and shouldn’t collide by cron/time alone: - 0 10 * * 1-5 (UTC) - 0 7-23 * * 1-5 (America/New_York) - 0 */6 * * 0,6 (America/New_York) What we know so far - I don’t see a confirmed, open 3.6.16 bug that exactly matches “three schedules stop producing upcoming runs until toggled”. However, there has been recent work around multiple schedules and schedule slug handling. I want to dig deeper with a repro if you’re open to it. - There’s a somewhat-related prior report about multiple schedules not producing upcoming runs when they share the same cron (older 3.x), which your setup does not, but the symptoms are similar: Multiple schedules with the same cron expression are not properly executed. - 3.6.17 release notes added schedule slug rename support (“replaces”): PR #20460 and the 3.6.16→3.6.17 diff is here: Full changelog. A few targeted questions to help us narrow it down - Prefect Cloud or self-hosted Prefect Server? - Did all three schedules stall, or just one of them? Roughly when did “Upcoming Runs” stop advancing (timestamp + timezone)? - Did you recently rename schedules (or change schedule names) before this occurred? - After toggling off/on, did all three schedules begin producing runs again? Ways to verify timezones and schedule state - Python client (shows full schedule objects including timezone):
Copy code
import anyio
  from prefect.client.orchestration import get_client

  async def main(deployment_id: str):
      async with get_client() as client:
          schedules = await client.read_deployment_schedules(deployment_id)
          for s in schedules:
              print(s.id, s.schedule, getattr(s.schedule, "timezone", None), s.active)

  anyio.run(main, "<your-deployment-id>")
- CLI (lists schedules, though timezone isn’t shown):
Copy code
prefect deployment schedule ls "<flow-name>/<deployment-name>" -o json
- Toggle (your current workaround, confirmed to “reset” upcoming runs):
Copy code
prefect deployment schedule pause "<flow-name>/<deployment-name>"
  prefect deployment schedule resume "<flow-name>/<deployment-name>"
Short-term mitigations to try - Keep all three schedules active but stagger by a minute to rule out any edge-case de-duplication: - e.g., change one to run at minute 1 or 2 past the hour temporarily and see if the stall recurs. - If you renamed schedules recently, avoid chained/swap renames for now; keep names/slugs unique and change them one at a time. If you can share: - Cloud vs self-hosted - The last “good” upcoming run timestamp and the first “missing” one - Whether only one schedule stalled or all three …I can open a GitHub issue with a minimal repro using your cron expressions and track it with the team. If you’d prefer to file it directly, we’ll jump on it — please include your YAML schedules and the “toggling schedules restored upcoming runs” detail.
k
@Marvin this is prefect self-hosted on AWS ECS Fargate. toggling off and on generated the next 3 scheduled runs but did not auto schedule additional ones. reverting back to 3.4.24 seemingly fixed the issue.
m
thought for 166.7 seconds
Got it — thanks for the extra detail. What you’re seeing on 3.6.16 (self‑hosted on ECS Fargate) matches a regression we’ve identified with multi‑schedule deployments: the scheduler will generate exactly the “minimum” number of future runs (3) and then stop advancing for one or more schedules. Toggling off/on produces another 3, but it won’t keep auto‑scheduling beyond that window. Why it happens (summary) - In 3.6.16, the scheduler’s “do I need to create more runs?” check was using an aggregate across all schedules in a deployment instead of checking each schedule independently. - With multiple schedules (especially when they have different frequencies), one schedule can be “starved” while the aggregate count still looks sufficient — so the deployment isn’t picked again and upcoming runs stop advancing for that schedule. - This isn’t caused by timezones; it’s selection logic. Status - The bug is fixed on main with per‑schedule checks and will be included in the next 3.6.x release (not in 3.6.17). Commit reference: Fix scheduler to check per‑schedule run counts for multi‑schedule deployments. - Your observation that reverting to 3.4.24 restores normal behavior makes sense; the regression was introduced in the 3.6.x line. Workarounds until the fix is released - Keep your current rollback to 3.4.24, or - Split the three schedules into separate deployments (most reliable workaround on 3.6.16/3.6.17), or - As a stopgap, you can continue toggling schedules, but you’ll only get another batch of ~3 each time by design. Increasing the scheduler’s “min upcoming runs” buffer (if you’ve customized server settings) can extend that window, but it won’t resolve the underlying selection issue. Notes on your dump - The CLI JSON output intentionally flattens the schedule to a string (e.g.,
cron: 0 7-23 * * 1-5
) and drops timezone. The API does store your timezones (UTC and America/New_York) as you defined in prefect.yaml; to see them programmatically, use the client/REST API instead of
-o json
. If you’d like, I can track this for you and ping when the patch release is live; otherwise, once 3.6.18 (or the 3.6.x release that includes the fix) is out, upgrading should resolve it without further changes to your schedules.
k
@Marvin is there a github issue for this bug?
m
thought for 38.1 seconds
Short answer: I can’t find an existing public GitHub issue that tracks this exact 3.6.16 regression (multi-schedule deployments only generating ~3 future runs, then stalling). I also don’t see a matching discussion. If you’d like, I can open an issue on your behalf with the details you’ve shared. I’d include: - Environment: self-hosted Prefect Server on AWS ECS Fargate - Version regression: works on 3.4.24, breaks on 3.6.16 - Repro schedules: - 0 10 * * 1-5 (UTC) - 0 7-23 * * 1-5 (America/New_York) - 0 */6 * * 0,6 (America/New_York) - Symptoms: Upcoming Runs stop advancing; toggling schedules creates the next 3 only; auto-scheduling then stops again - Any recent schedule renames (if applicable) Alternatively, you can file it directly here: Open a new issue — and I’ll make sure it gets routed quickly. Want me to create the issue for you? If so, please confirm: - Exact Prefect Server version strings for 3.6.16 and 3.4.24 (
prefect version
) - Whether any schedule names/slugs were renamed recently - Approximate timestamp/timezone when Upcoming Runs stopped advancing - Whether all three schedules stalled, or just one/two
k
@Marvin yes please create an issue for this and link to this chat if it helps. • i dont have the full version string for 3.6.16 anymore because i rolled back • no names or slugs were renamed recently • last scheduled run was monday feb 16 9am EST. the schedule is supposed to run hourly on mon-fri between 7am and 11pm. no runs were generated after 9am. toggling off and on at 3pm EST, it generated runs for 4,5 and 6pm, but no runs after that. rolling back to 3.4.24 rsolved it.
m
Error: GraphQL errors: [{'type': 'FORBIDDEN', 'path': ['repository'], 'extensions': {'saml_failure': False}, 'locations': [{'line': 3, 'column': 13}], 'message': "The 'PrefectHQ' organization forbids access via a personal access tokens (classic) if the token's lifetime is greater than 180 days. Please adjust your token's lifetime at the following URL: https://github.com/settings/tokens/2034986401"}]
Sorry, I encountered an error while processing your request. Please try again.