Rasmus Simmelkjær Berntsen
11/14/2025, 12:07 PMMarvin
11/14/2025, 12:07 PMMarvin
11/14/2025, 12:15 PMPREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME (default PT1H)
- PREFECT_SERVER_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME (default P100D)
- Docs: Schedules and Settings reference
Recommended maintenance approach
1) Before upgrade
- Pause all work pools (quick and global):
prefect work-pool ls
prefect work-pool pause "<POOL_NAME>"
Verified commands:
- prefect work-pool pause <NAME>
- prefect work-pool resume <NAME>
- prefect work-pool set-concurrency-limit <NAME> <LIMIT>
Docs: Work pools
- Optionally stop your workers. Pausing pools is usually enough: workers can keep running but won’t pick up new work while pools are paused.
- If you’re fully stopping the Prefect server, the scheduler won’t create any runs during downtime, so you won’t get a backlog from that period.
Optional: prevent even a small number of Scheduled runs from being created while the server stays up
- If you’re leaving the server up during maintenance and want zero new scheduled runs, you can pause deployment schedules. This is more granular but more work if you have many deployments:
prefect deployment schedule ls "<DEPLOYMENT_NAME>"
prefect deployment schedule pause "<DEPLOYMENT_NAME>" <SCHEDULE_ID>
You’ll resume with:
prefect deployment schedule resume "<DEPLOYMENT_NAME>" <SCHEDULE_ID>
API if you want to script it across many deployments: Pause and Resume
2) After upgrade
- Start the server, then resume pools. To avoid a spike, resume with a low concurrency limit and scale up:
prefect work-pool set-concurrency-limit "<POOL_NAME>" 1
prefect work-pool resume "<POOL_NAME>"
# Then gradually increase:
prefect work-pool set-concurrency-limit "<POOL_NAME>" 5
prefect work-pool set-concurrency-limit "<POOL_NAME>" 20
- If you paused deployment schedules, resume them as needed.
Why this avoids “spamming”
- If the server was down, there’s no catch-up backlog created by the scheduler.
- If pools were paused while the server was up, the scheduler may have created only a handful of future runs (limited by the scheduler’s min/max window). Those will start when you resume pools, and you can throttle with work pool concurrency limits.
Notes
- Pausing a work pool does not cancel runs already in progress—just stops new work from being picked up.
- If you truly need zero new run objects created while the server remains up, pause deployment schedules (or stop the scheduler by stopping the server).
Would you prefer a short script to pause/resume all pools in your environment, or do you want to pause deployment schedules across many deployments via the API?Rasmus Simmelkjær Berntsen
11/14/2025, 12:30 PMMarvin
11/14/2025, 12:30 PMMarvin
11/14/2025, 12:32 PM# Pause all pools you use
prefect work-pool pause "<POOL_NAME>"
# After maintenance, throttle then resume
prefect work-pool set-concurrency-limit "<POOL_NAME>" 1
prefect work-pool resume "<POOL_NAME>"
# Then gradually increase
prefect work-pool set-concurrency-limit "<POOL_NAME>" 5
prefect work-pool set-concurrency-limit "<POOL_NAME>" 20
Docs: Work pools
If you must take the Prefect server down
- The scheduler does not backfill missed runs automatically. To ensure “missed” runs are created to run afterward, backfill via the deployment scheduling API once the server is back:
- REST: POST /deployments/{id}/schedule
- Send start_time and end_time covering your maintenance window (in the past) to create the runs, then let pool concurrency limits throttle execution.
Example:
# Replace variables appropriately
export PREFECT_API_URL="http://<your-server>/api"
export DEPLOYMENT_ID="<uuid>"
curl -X POST "$PREFECT_API_URL/deployments/$DEPLOYMENT_ID/schedule" \
-H "Content-Type: application/json" \
-d '{
"start_time": "2025-11-14T09:00:00Z",
"end_time": "2025-11-14T12:00:00Z"
}'
Why this works
- With pools paused (server up), schedules still create run objects. They’ll queue in Scheduled state and execute after you resume, and your pool concurrency limit prevents flooding.
- With the server down, no runs are created during that time; the backfill API creates them after the fact. Then use pool concurrency to control throughput.
Related docs:
- Schedules: Concepts: Schedules
- Scheduler settings (look-ahead windows): Settings referenceRasmus Simmelkjær Berntsen
11/14/2025, 12:58 PMMarvin
11/14/2025, 12:58 PMMarvin
11/14/2025, 1:01 PMmax_runs=2 to cap how many are created.
- Use timezone-aware timestamps that match your deployment’s timezone.
REST ref: POST /deployments/{id}/schedule