<@ULVA73B9P>
I have two deployments which run dail...
# ask-marvinp
Pav Staz
03/09/2026, 1:37 PM@Marvin
I have two deployments which run daily in the early hours. I set the cron time for 1 for 00:30 and one for 4am.
It even states the correct time (i.e. 04:00 everyday) next to each deployment in the UI, and the deployments have been successfully triggering at these times for over 2 weeks!
_pre_dbt_syncs_._serve_(
name="pre_dbt_syncs",
cron="30 0 * * *"
)
_post_dbt_jobs_._serve_(
name="post_dbt_jobs",
cron="0 4 * * *"
)
Setup:
docker container running the server, then a docker container for each deployment
The problem:
pre_dbt_syncs last ran on 07/03/2026 at 00:30 while the post_dbt_jobs ran last at 04:00 on the 06/03/2026.
In the UI, under the deployment, I do not see the next run scheduled at all
What I have checked:
Container logs for each deployment -> no errors and have been running for 4 days before incident
Server container logs - again no errors and been running for 4 days before the incident
In the server UI - Past run logs for these offending deployments - no errors
In the server UI - Other deployments schedules ran and I can see the next scheduled run for them
Prefect service stack restarts - 4 days before the incident
Host issues - non reported, other services have no errors and no CPU or RAM spikes
Edits to the deployment/server files - 2 weeks ago (and deployments have been running this long up until this point)
Where should I look to troubleshoot? what is the most likely cause?
I'm running prefect 3.6.19
m
Marvin
03/09/2026, 1:37 PM✅ thought for 595.1 seconds
Marvin
03/09/2026, 1:47 PMThanks for all the detail — that helps a lot.
Most likely cause
Given you’re using Flow.serve() in separate containers, the most common cause that matches your symptoms is: the serve runner paused the deployment schedules on a prior shutdown (default behavior), so the server stopped generating new runs. The already-generated “future” runs kept executing for a few days, then dried up — which is why you now see no “next run” in the UI and the last executions were 06–07 Mar.
Why this happens
- In Prefect 3.x, Flow.serve creates an API-managed deployment and server-side schedules.
- By default, serve pauses its deployment schedules on shutdown (
pause_on_shutdown=True Copy code
prefect deployment schedule ls pre_dbt_syncs/pre_dbt_syncs
prefect deployment schedule ls post_dbt_jobs/post_dbt_jobs Copy code
prefect deployment ls Copy code
prefect deployment inspect pre_dbt_syncs/pre_dbt_syncs
prefect deployment inspect post_dbt_jobs/post_dbt_jobs Copy code
prefect deployment schedule resume pre_dbt_syncs/pre_dbt_syncs --all
prefect deployment schedule resume post_dbt_jobs/post_dbt_jobs --all Copy code
pre_dbt_syncs.serve(
name="pre_dbt_syncs",
cron="30 0 * * *",
pause_on_shutdown=False
)
post_dbt_jobs.serve(
name="post_dbt_jobs",
cron="0 4 * * *",
pause_on_shutdown=False
)resume_deploymentprefect deployment schedule resume ... --allprefect deployment schedule ls ...date -uprefect deployment schedule ls <name>prefect deployment inspect <name>pause_on_shutdown=Truep
Pav Staz
03/09/2026, 3:51 PM@Marvin here is the output of the inspect and schedule commands
prefect deployment inspect 'pre-dbt-syncs/pre_dbt_syncs'
{
'id': 'cba46cab-45d7-444a-a9cd-56fcf1ad148f',
'created': '2026-02-20T170540.543844Z',
'updated': '2026-03-02T155103.613000Z',
'name': 'pre_dbt_syncs',
'version': 'd34bb2278b2f55fbce75e72209afbe26',
'version_id': None,
'version_info': None,
'branch': None,
'base': None,
'root': None,
'description': None,
'flow_id': '65570266-d027-4f05-b958-9951ae8eb724',
'concurrency_limit': None,
'global_concurrency_limit': None,
'concurrency_options': None,
'paused': False,
'schedules': [
{
'id': 'dd90ed2b-6e0a-4c2a-adbb-64f14817772e',
'created': '2026-03-02T155103.677105Z',
'updated': '2026-03-02T155103.677124Z',
'deployment_id': 'cba46cab-45d7-444a-a9cd-56fcf1ad148f',
'schedule': {'cron': '30 0 * * *', 'timezone': None, 'day_or': True},
'active': True,
'max_scheduled_runs': None,
'parameters': {},
'slug': None
}
],
'job_variables': {},
'parameters': {},
'pull_steps': None,
'tags': [],
'labels': {'prefect.flow.id': '65570266-d027-4f05-b958-9951ae8eb724'},
'work_queue_name': None,
'last_polled': '2026-03-04T003219.074355Z',
'parameter_openapi_schema': {'title': 'Parameters', 'type': 'object', 'properties': {}},
'path': '.',
'entrypoint': 'deployments/iit_pre_dbt_syncs.py:pre_dbt_syncs',
'storage_document_id': None,
'infrastructure_document_id': None,
'created_by': None,
'updated_by': None,
'work_queue_id': None,
'enforce_parameter_schema': True,
'work_pool_name': None,
'status': 'READY',
'automations': []
}
root@fa6a1e05a279:/opt/project# prefect deployment inspect 'post-dbt-jobs/post_dbt_jobs'
{
'id': 'ad0ef257-4669-454a-8458-5adbab012d0f',
'created': '2026-02-20T170540.169718Z',
'updated': '2026-03-02T155057.701000Z',
'name': 'post_dbt_jobs',
'version': '5517ada88c6167ea94157e168f8cbeee',
'version_id': None,
'version_info': None,
'branch': None,
'base': None,
'root': None,
'description': None,
'flow_id': '357b8a05-2b3f-40ec-9efc-f005f62d01d1',
'concurrency_limit': None,
'global_concurrency_limit': None,
'concurrency_options': None,
'paused': False,
'schedules': [
{
'id': '084cf54b-918b-4efc-bf78-69013402f169',
'created': '2026-03-02T155057.747228Z',
'updated': '2026-03-02T155057.747248Z',
'deployment_id': 'ad0ef257-4669-454a-8458-5adbab012d0f',
'schedule': {'cron': '0 4 * * *', 'timezone': None, 'day_or': True},
'active': True,
'max_scheduled_runs': None,
'parameters': {},
'slug': None
}
],
'job_variables': {},
'parameters': {},
'pull_steps': None,
'tags': [],
'labels': {'prefect.flow.id': '357b8a05-2b3f-40ec-9efc-f005f62d01d1'},
'work_queue_name': None,
'last_polled': '2026-03-04T002942.794769Z',
'parameter_openapi_schema': {'title': 'Parameters', 'type': 'object', 'properties': {}},
'path': '.',
'entrypoint': 'deployments/iit_post_dbt_jobs.py:post_dbt_jobs',
'storage_document_id': None,
'infrastructure_document_id': None,
'created_by': None,
'updated_by': None,
'work_queue_id': None,
'enforce_parameter_schema': True,
'work_pool_name': None,
'status': 'READY',
'automations': []
}
root@fa6a1e05a279:/opt/project# prefect deployment schedule ls 'pre-dbt-syncs/pre_dbt_syncs'
Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID ┃ Schedule ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ dd90ed2b-6e0a-4c2a-adbb-64f14817772e │ cron: 30 0 * * * │ True │
└──────────────────────────────────────┴──────────────────┴────────┘
root@fa6a1e05a279:/opt/project# prefect deployment schedule ls 'post-dbt-jobs/post_dbt_jobs'
Deployment Schedules
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ ID ┃ Schedule ┃ Active ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ 084cf54b-918b-4efc-bf78-69013402f169 │ cron: 0 4 * * * │ True │
└──────────────────────────────────────┴─────────────────┴────────┘
root@fa6a1e05a279:/opt/project#
m
Marvin
03/09/2026, 3:51 PM✅ thought for 236.6 seconds
Marvin
03/09/2026, 3:55 PMThanks for sharing those outputs — a few key observations:
- Both deployments are not paused and their schedules are active.
- The UI shows no “Next run”, which only happens when there are no future SCHEDULED flow runs in the DB for that deployment.
- Both deployments show last_polled around 2026-03-04 — that’s stale. It suggests the serve runner may not be polling these deployments anymore (or that field isn’t being updated by your runner), but importantly: the “Next run” in the UI does not depend on the runner — it depends on the server-side scheduler generating future runs. So the scheduler is not generating runs for just these two schedules right now.
Fastest next steps to confirm and fix
1) Confirm there are truly no future scheduled runs
- This will tell us if the scheduler has stopped generating for these deployments.
Copy code
prefect flow-run ls --flow-name pre_dbt_syncs --state SCHEDULED --limit 10
prefect flow-run ls --flow-name post_dbt_jobs --state SCHEDULED --limit 10 Copy code
prefect deployment schedule pause 'pre-dbt-syncs/pre_dbt_syncs' --all
prefect deployment schedule resume 'pre-dbt-syncs/pre_dbt_syncs' --all
prefect deployment schedule pause 'post-dbt-jobs/post_dbt_jobs' --all
prefect deployment schedule resume 'post-dbt-jobs/post_dbt_jobs' --allRunnerDeployment.apply() Copy code
prefect deployment schedule create 'pre-dbt-syncs/pre_dbt_syncs' --cron "*/5 * * * *"
# wait ~1–2 minutes and check UI, then:
prefect deployment schedule delete 'pre-dbt-syncs/pre_dbt_syncs' <that_new_schedule_id>pause_on_shutdown=False Copy code
pre_dbt_syncs.serve(name="pre_dbt_syncs", cron="30 0 * * *", pause_on_shutdown=False)
post_dbt_jobs.serve(name="post_dbt_jobs", cron="0 4 * * *", pause_on_shutdown=False)flow-run lsMarvin
03/09/2026, 3:55 PMIf that doesn’t resolve it, we’ll dig into the server scheduler logs next.
p
Pav Staz
03/09/2026, 4:23 PM@Marvin I'm unsure if it is the pause_on_shutdown hypothesis since the inspect command of the deployment showed that it was active and the schedule was True.
Last polled being 3 days before the last run seems to me that something went wrong there.
I toggled my deployment schedule off and on. After waiting a few minutes the UI did not create any scheduled runs under 'Upcoming'. Perhaps the creation of scheduled runs relies on polling isn't that frequent? I also did this toggle on a deployment that had a working schedule and they also did not materialise.
I believe I have sorted the problem, and i've just restarted the whole stack of prefect server and deployments, now I see the next 3 scheduled runs under 'upcoming' on each deployment.
My next question is: "Are the maximum number of scheduled runs only ever 3? and when the deployment polls the scheduler do more appear on a 3-run rolling basis?"
m
Marvin
03/09/2026, 4:23 PM✅ thought for 118.3 seconds
Marvin
03/09/2026, 4:25 PMGreat to hear a full restart brought “Upcoming” back — that strongly points to the server-side scheduler state getting “stuck” and being reset on restart.
Answer to your question on “3 upcoming runs”
- By default, Prefect’s server-side scheduler targets at least 3 upcoming auto-scheduled runs per schedule, and no more than 100 runs per batch, while also staying within a time horizon.
- So yes: seeing 3 in Upcoming is expected with defaults. As time passes (and as the scheduler loop runs every ~60s by default), it keeps that window “topped up” — effectively a rolling basis.
Defaults that control this behavior
- min_runs = 3
- max_runs = 100
- min_scheduled_time = PT1H (1 hour)
- max_scheduled_time = P100D (100 days)
- loop_seconds = 60
- recent_deployments_loop_seconds = 5
How it works
- The server’s scheduler periodically checks each active schedule. If there are fewer than min_runs future SCHEDULED flow runs, or if the furthest SCHEDULED run is sooner than min_scheduled_time from “now”, it generates more.
- It won’t schedule beyond max_scheduled_time into the future, and it won’t make more than max_runs in one go.
- The runner/serve process does not create the schedule — it only polls for and executes runs that the server already created. The “Upcoming” list in the UI strictly reflects what the server scheduled.
If you want to see more than 3 in Upcoming
- Increase min_runs and/or min_scheduled_time on the server. You can set these via env vars or your Prefect config:
- PREFECT_SERVER_SERVICES_SCHEDULER_MIN_RUNS
- PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME
- PREFECT_SERVER_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME
- Example (env):
- PREFECT_SERVER_SERVICES_SCHEDULER_MIN_RUNS=10
- PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME=PT6H
- Then restart the server so the scheduler picks up the new settings.
- Verify your current values:
Copy code
prefect config view --show-defaults --show-sourcesprefect config view --show-defaults --show-sources | grep -i scheduler -n7 Views