<@ULVA73B9P> I am self hosting a prefect server - ...
# ask-marvin
r
@Marvin I am self hosting a prefect server - If i want to pause all my deployments (eg if I need to do an upgrade to the prefect server), what is the best way of doing so? Can i pause the work pool(s)? And when i resume it again after the upgrade, how do I make sure that my server isnt spammed with the runs that needs to catch up?
m
thought for 502.1 seconds
Great question. Assuming Prefect 3.x, here’s a safe, simple maintenance workflow and how to avoid a post-upgrade flood of runs. What to know about “catch-up” - The Prefect scheduler does not backfill missed runs during server downtime. When your server is down, no runs are created; when it comes back up, it schedules from “now” forward within a small look-ahead window. This prevents a flood. - That window is controlled by settings like: -
PREFECT_SERVER_SERVICES_SCHEDULER_MIN_SCHEDULED_TIME
(default PT1H) -
PREFECT_SERVER_SERVICES_SCHEDULER_MAX_SCHEDULED_TIME
(default P100D) - Docs: Schedules and Settings reference Recommended maintenance approach 1) Before upgrade - Pause all work pools (quick and global):
Copy code
prefect work-pool ls
  prefect work-pool pause "<POOL_NAME>"
Verified commands: -
prefect work-pool pause <NAME>
-
prefect work-pool resume <NAME>
-
prefect work-pool set-concurrency-limit <NAME> <LIMIT>
Docs: Work pools - Optionally stop your workers. Pausing pools is usually enough: workers can keep running but won’t pick up new work while pools are paused. - If you’re fully stopping the Prefect server, the scheduler won’t create any runs during downtime, so you won’t get a backlog from that period. Optional: prevent even a small number of Scheduled runs from being created while the server stays up - If you’re leaving the server up during maintenance and want zero new scheduled runs, you can pause deployment schedules. This is more granular but more work if you have many deployments:
Copy code
prefect deployment schedule ls "<DEPLOYMENT_NAME>"
  prefect deployment schedule pause "<DEPLOYMENT_NAME>" <SCHEDULE_ID>
You’ll resume with:
Copy code
prefect deployment schedule resume "<DEPLOYMENT_NAME>" <SCHEDULE_ID>
API if you want to script it across many deployments: Pause and Resume 2) After upgrade - Start the server, then resume pools. To avoid a spike, resume with a low concurrency limit and scale up:
Copy code
prefect work-pool set-concurrency-limit "<POOL_NAME>" 1
  prefect work-pool resume "<POOL_NAME>"
  # Then gradually increase:
  prefect work-pool set-concurrency-limit "<POOL_NAME>" 5
  prefect work-pool set-concurrency-limit "<POOL_NAME>" 20
- If you paused deployment schedules, resume them as needed. Why this avoids “spamming” - If the server was down, there’s no catch-up backlog created by the scheduler. - If pools were paused while the server was up, the scheduler may have created only a handful of future runs (limited by the scheduler’s min/max window). Those will start when you resume pools, and you can throttle with work pool concurrency limits. Notes - Pausing a work pool does not cancel runs already in progress—just stops new work from being picked up. - If you truly need zero new run objects created while the server remains up, pause deployment schedules (or stop the scheduler by stopping the server). Would you prefer a short script to pause/resume all pools in your environment, or do you want to pause deployment schedules across many deployments via the API?
r
@Marvin - How can I make sure that runs that we scheduled to run during a maintenance windows, just runs afterwards? (Then I would enforce a concurrency limit to avoid flooding)
m
thought for 114.4 seconds
Short answer: leave schedules active and pause execution. If you want runs that “would have” started during a maintenance window to run afterwards, do NOT pause deployment schedules. Instead: - Pause your work pools (or stop workers) before maintenance - Leave the server and schedules up so runs get created in Scheduled state - After maintenance, resume pools with a low concurrency limit and scale up Commands (verified):
Copy code
# Pause all pools you use
prefect work-pool pause "<POOL_NAME>"

# After maintenance, throttle then resume
prefect work-pool set-concurrency-limit "<POOL_NAME>" 1
prefect work-pool resume "<POOL_NAME>"
# Then gradually increase
prefect work-pool set-concurrency-limit "<POOL_NAME>" 5
prefect work-pool set-concurrency-limit "<POOL_NAME>" 20
Docs: Work pools If you must take the Prefect server down - The scheduler does not backfill missed runs automatically. To ensure “missed” runs are created to run afterward, backfill via the deployment scheduling API once the server is back: - REST: POST /deployments/{id}/schedule - Send
start_time
and
end_time
covering your maintenance window (in the past) to create the runs, then let pool concurrency limits throttle execution. Example:
Copy code
# Replace variables appropriately
export PREFECT_API_URL="http://<your-server>/api"
export DEPLOYMENT_ID="<uuid>"

curl -X POST "$PREFECT_API_URL/deployments/$DEPLOYMENT_ID/schedule" \
  -H "Content-Type: application/json" \
  -d '{
    "start_time": "2025-11-14T09:00:00Z",
    "end_time": "2025-11-14T12:00:00Z"
  }'
Why this works - With pools paused (server up), schedules still create run objects. They’ll queue in Scheduled state and execute after you resume, and your pool concurrency limit prevents flooding. - With the server down, no runs are created during that time; the backfill API creates them after the fact. Then use pool concurrency to control throughput. Related docs: - Schedules: Concepts: Schedules - Scheduler settings (look-ahead windows): Settings reference
r
@Marvin - So using the backfill api for a deployment that runs every at hh:00, hh:15, hh:30 and hh:45, and I have a maintenance window form say hh:12 to hh:32, and I use the backfill api with start_time to hh:11 to hh:31, it will schedule the runs that should have run ah hh:15 and hh:30?
m
thought for 186.3 seconds
Yes. If your deployment runs at hh:00, hh:15, hh:30, hh:45 and you call the backfill API with: - start_time = hh:11 - end_time = hh:31 it will create the occurrences at hh:15 and hh:30 (and not hh:00 or hh:45). In Prefect 3, the start_time is inclusive and end_time acts as an upper bound, so everything scheduled between those times will be generated. Tips to be extra safe: - Set a small buffer on the end (e.g., hh:32) or specify
max_runs=2
to cap how many are created. - Use timezone-aware timestamps that match your deployment’s timezone. REST ref: POST /deployments/{id}/schedule