Yes — but the best way depends on what you want to stop. In Prefect 3.x, there isn’t a single built-in “maintenance mode” switch for all deployments across a workspace. But you do have a few practical options: Best options 1. Pause deployment schedules This stops new scheduled runs from being created for those deployments. - CLI support exists via

prefect deployment pause_schedule

- There’s also

resume_schedule

afterward 2. Pause the work pool or work queue This prevents workers from picking up new work from that pool/queue. - CLI support exists via: -

prefect work-pool pause

-

prefect work-queue pause

Which one should you use? - If you want to stop new runs from being scheduled at all: pause deployment schedules - If you want to stop execution during the window, regardless of how runs got created: pause the work pool or work queue - If you want to be safest during maintenance: do both Important nuance These are not exactly the same: - Pausing a deployment schedule prevents future scheduled flow runs from being generated - Pausing a work pool/queue means runs may still exist in a scheduled/late state, but workers won’t start them until you resume So for a maintenance window, a common pattern is: 1. Pause the relevant work pool/queue shortly before maintenance 2. Optionally pause deployment schedules too, if you don’t want backlog created 3. Resume both afterward CLI commands available in Prefect 3.x I verified these command groups exist in the CLI: -

prefect deployment

includes

pause_schedule

/

resume_schedule

-

prefect work-pool

includes

pause

/

resume

-

prefect work-queue

includes

pause

/

resume

Practical recommendation If all of your deployments run through the same work pool, the easiest approach is usually:

prefect work-pool pause <POOL_NAME>

Then after maintenance:

prefect work-pool resume <POOL_NAME>

If you also want to stop schedules from accumulating runs, pause each deployment’s schedule too. What’s missing There does not appear to be a native one-click global “pause all deployments in workspace until X time” feature. So if you truly mean all deployments, you’d likely automate it with a small script using the Prefect API: - list deployments, then pause each schedule - and/or pause the work pools they use If you want, I can help you with either of these next: - a Python script to pause/resume all deployment schedules - a CLI-based approach for all work pools/queues - a maintenance-window automation flow that pauses everything and resumes later

That makes sense — if the control plane itself may be unavailable, pausing the work pool is not the most robust option, because the worker still has to keep polling Prefect and can fail while “phoning home.” In your case, the safer maintenance pattern is: Stop the workers entirely If you run workers under

systemd

, shutting down the worker processes is the most reliable way to prevent new runs from starting during an outage window. Recommended approach 1. Before maintenance - stop the

systemd

service(s) running your Prefect workers - optionally pause deployment schedules too, if you also want to avoid a backlog of scheduled runs being created 2. After maintenance - start the worker services again - resume schedules if you paused them Why this is safer - No worker process is alive to poll the API - No worker-side errors from failed health/pool checks - No chance of the worker grabbing work right before or during the outage Tradeoff If you only stop workers and do not pause schedules: - scheduled flow runs may still be created by Prefect - they’ll wait until workers come back - after restart, workers may pick up that backlog If you want no execution during outage but are okay with catch-up afterward, stop workers only is enough. If you want no execution and minimal backlog, do both: - stop workers - pause deployment schedules Suggested systemd pattern If your worker is managed like this:

sudo systemctl stop prefect-worker

and later:

sudo systemctl start prefect-worker

that’s a cleaner maintenance boundary than relying on pool pause state. If you have multiple worker services, you could stop them all with a naming convention, for example via a shell script. Extra caution If a worker is already executing a flow run when you stop it: - that run may be interrupted depending on your infrastructure type - for subprocess/process workers, it may terminate the run - for external infrastructure, behavior depends on where the flow actually runs So if you need a graceful drain, you may want to: 1. stop scheduling first 2. wait for active runs to finish 3. stop workers Good maintenance strategies Fast hard stop - stop all worker services - do maintenance - restart services Graceful drain - pause deployment schedules - wait for in-flight runs to complete - stop worker services - do maintenance - restart worker services - resume schedules Given the traceback you saw in

_check_work_pool_paused

, I agree with your conclusion: don’t rely on “paused but still connected” workers when the outage affects API connectivity. If you want, I can help you with: - a systemd unit/template pattern for running multiple Prefect workers safely - a pre-maintenance shell script to stop all workers and optionally pause schedules - a drain script that checks for active flow runs before stopping workers

Absolutely — and yes, you can usually consolidate. If your current

systemd

units are one deployment apiece, that often means you may be running

prefect flow serve

-style processes or otherwise tying one long-lived process to one deployment. In Prefect 3.x, the more scalable pattern is usually: - many deployments - routed to one work pool / work queue - serviced by one worker process per infrastructure type / environment So instead of: - 20 deployments - 20 systemd services you often want: - 20 deployments - 1 work pool - 1 worker service or maybe a small number of workers split by: - environment - infra type - resource class - team boundary That’s the first thing I’d recommend. Important distinction: If you are truly running workers, a single worker can pick up work for many deployments in the same pool/queue. If you are running served flows (

flow.serve()

/

prefect flow serve

), that is inherently much closer to one process per served flow, and consolidation means moving to deployments + workers. So if you want, later I can help you map your current setup to one of these two patterns: - consolidated worker model for deployments - served-flow model if that’s what you actually have today --- Recommended consolidated systemd pattern A good pattern is: - one templated

systemd

unit for workers - each instance points at a work pool - optionally one instance per queue / profile / environment Example:

# /etc/systemd/system/prefect-worker@.service
[Unit]
Description=Prefect Worker (%i)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=prefect
Group=prefect
WorkingDirectory=/opt/prefect
Environment=PREFECT_PROFILE=production
ExecStart=/usr/local/bin/prefect worker start --pool %i
Restart=on-failure
RestartSec=15s

# Quick retries are handled by systemd automatically with RestartSec.
# Allow unlimited restart attempts over time instead of failing permanently.
StartLimitIntervalSec=0

# Optional shutdown tuning
TimeoutStopSec=120
KillMode=mixed

[Install]
WantedBy=multi-user.target

Then manage instances like:

sudo systemctl enable --now prefect-worker@my-pool
sudo systemctl stop prefect-worker@my-pool
sudo systemctl start prefect-worker@my-pool
sudo systemctl restart prefect-worker@my-pool

Why this helps - one service definition, many instances - much easier maintenance - no hardcoded per-deployment services - easier outage handling --- About your retry requirement You mentioned: - today it fails hard after 3 retries - during outages you want quick retries first, then at least hourly retries indefinitely That is a good fit for

systemd

. Instead of capping retries permanently, use: -

Restart=on-failure

-

RestartSec=

for the base retry delay -

StartLimitIntervalSec=0

to avoid permanent lockout That gives you retry forever at a fixed interval. If you want quick retries first, then hourly retries,

systemd

alone does not do stepped backoff especially elegantly. The cleanest way is one of these: 1. Simple and robust: retry every hour always 2. Moderately aggressive: retry every 5–15 minutes always 3. Advanced: wrap

prefect worker start

in a shell script that implements staged backoff Because you explicitly want: - quick retries - then slow hourly retries I’d recommend a wrapper script. Example:

#!/usr/bin/env bash
set -euo pipefail

POOL_NAME="${1:?pool name required}"
PROFILE="${PREFECT_PROFILE:-production}"

attempt=0

while true; do
  attempt=$((attempt + 1))
  echo "$(date -Is) starting Prefect worker for pool: ${POOL_NAME}, attempt ${attempt}"

  prefect --profile "${PROFILE}" worker start --pool "${POOL_NAME}"
  exit_code=$?

  echo "$(date -Is) worker exited with code ${exit_code}"

  # Quick retries first
  if [ "${attempt}" -le 3 ]; then
    sleep 60
  else
    # Then settle into hourly retries
    sleep 3600
  fi
done

And the unit becomes: ``` [Unit]

Description=Prefect Worker (%i) After=network-online.target Wants=network-online.target [Service] Type=simple User=prefect Group=prefect WorkingDirectory=/opt/prefect Environment=PREFECT_PROFILE=production ExecStart=/opt/prefect/bin/run-prefect-worker.sh %i Restart=always RestartSec=5s StartLimitIntervalSec=0 TimeoutStopSec=120 KillMode=mixed [Install] WantedBy=multi-user.target

*Note:* if the wrapper itself loops forever, `Restart=always` is mostly just insurance in case the script itself dies.

---

*Maintenance script #1: pre-maintenance stop workers + optionally pause schedules*

Here’s a draft shell script that:

- stops one or more `systemd` worker units
- optionally pauses specified deployments

Because I could not verify the exact CLI flags for `pause_schedule` in-tool due command timeouts, I’m keeping the deployment pause section clearly marked for you to align with your local CLI help.

#!/usr/bin/env bash set -euo pipefail # Example usage: # ./prefect-maintenance-down.sh # ./prefect-maintenance-down.sh --pause-schedules PAUSE_SCHEDULES=false if [[ "${1:-}" == "--pause-schedules" ]]; then PAUSE_SCHEDULES=true fi WORKER_UNITS=( "prefect-worker@my-pool.service" "prefect-worker@another-pool.service" ) # Use the deployment identifiers that match your environment. # Replace these with your actual deployment names. DEPLOYMENTS=( "flow-a/deployment-a" "flow-b/deployment-b" ) echo "Stopping Prefect worker services..." for unit in "${WORKER_UNITS[@]}"; do echo " stopping ${unit}" sudo systemctl stop "${unit}" done echo "Worker service state:" for unit in "${WORKER_UNITS[@]}"; do sudo systemctl is-active "${unit}" || true done if [[ "${PAUSE_SCHEDULES}" == "true" ]]; then echo "Pausing deployment schedules..." for dep in "${DEPLOYMENTS[@]}"; do echo " pausing ${dep}" prefect deployment pause_schedule "${dep}" done fi echo "Maintenance shutdown steps complete."

---

*Maintenance script #2: post-maintenance start workers + optionally resume schedules*

#!/usr/bin/env bash set -euo pipefail # Example usage: # ./prefect-maintenance-up.sh # ./prefect-maintenance-up.sh --resume-schedules RESUME_SCHEDULES=false if [[ "${1:-}" == "--resume-schedules" ]]; then RESUME_SCHEDULES=true fi WORKER_UNITS=( "prefect-worker@my-pool.service" "prefect-worker@another-pool.service" ) DEPLOYMENTS=( "flow-a/deployment-a" "flow-b/deployment-b" ) echo "Starting Prefect worker services..." for unit in "${WORKER_UNITS[@]}"; do echo " starting ${unit}" sudo systemctl start "${unit}" done echo "Worker service state:" for unit in "${WORKER_UNITS[@]}"; do sudo systemctl is-active "${unit}" done if [[ "${RESUME_SCHEDULES}" == "true" ]]; then echo "Resuming deployment schedules..." for dep in "${DEPLOYMENTS[@]}"; do echo " resuming ${dep}" prefect deployment resume_schedule "${dep}" done fi echo "Maintenance startup steps complete."

---

*Drain script draft: pause schedules, wait for active runs, then stop workers*

This one is a little trickier because the exact `prefect flow-run ls` filtering flags should really be verified locally first. Since my CLI verification timed out, I’ll give you a *safe draft* with a pluggable query section.

Conceptually it does:

1. pause schedules
2. poll for active flow runs
3. stop workers when zero remain

#!/usr/bin/env bash set -euo pipefail POLL_SECONDS="${POLL_SECONDS:-60}" TIMEOUT_SECONDS="${TIMEOUT_SECONDS:-7200}" WORKER_UNITS=( "prefect-worker@my-pool.service" ) DEPLOYMENTS=( "flow-a/deployment-a" "flow-b/deployment-b" ) start_ts="$(date +%s)" echo "Pausing deployment schedules..." for dep in "${DEPLOYMENTS[@]}"; do echo " pausing ${dep}" prefect deployment pause_schedule "${dep}" done echo "Waiting for active flow runs to drain..." while true; do now_ts="$(date +%s)" elapsed="$((now_ts - start_ts))" if (( elapsed > TIMEOUT_SECONDS )); then

echo "Timeout waiting for active flow runs to drain." exit 1 fi # Replace this command with the exact local CLI/API query you prefer. # The goal is to count RUNNING / PENDING / SCHEDULED runs that you consider "in-flight". active_count="$( prefect flow-run ls 2>/dev/null | grep -E 'Running|Pending|Late|Scheduled' | wc -l || true )" active_count="$(echo "${active_count}" | tr -d '[space]')" echo "$(date -Is) active-ish flow runs: ${active_count}" if [[ "${active_count}" == "0" ]]; then break fi sleep "${POLL_SECONDS}" done echo "No active flow runs detected. Stopping worker services..." for unit in "${WORKER_UNITS[@]}"; do sudo systemctl stop "${unit}" done echo "Drain complete."

---

*Better drain option: use the Python API instead of parsing CLI output*

This is the one I’d actually trust more in production. It can be made precise about:
- which deployments
- which work pool
- which states count as active

Draft:

#!/usr/bin/env python3 import subprocess import sys import time from datetime import datetime, timezone from prefect.client.orchestration import get_client WORKER_UNITS = [ "prefect-worker@my-pool.service", ] DEPLOYMENT_NAMES = [ "flow-a/deployment-a", "flow-b/deployment-b", ] POLL_SECONDS = 60 TIMEOUT_SECONDS = 7200 def run(cmd): print(f"{datetime.now(timezone.utc).isoformat()} running: {' '.join(cmd)}") subprocess.run(cmd, check=True) async def pause_schedules(client): # Placeholder: you may prefer doing this with CLI or by deployment IDs once mapped. for name in DEPLOYMENT_NAMES: run(["prefect", "deployment", "pause_schedule", name]) async def count_active_runs(client): # This should be refined to your exact state criteria and deployment filter. runs = await client.read_flow_runs(limit=200) active_types = {"RUNNING", "PENDING", "SCHEDULED", "LATE"} return sum( 1 for run in runs if getattr(getattr(run, "state", None), "type", None) and str(run.state.type).upper() in active_types ) async def main(): async with get_client() as client: await pause_schedules(client) started = time.time() while True: active = await count_active_runs(client) print(f"{datetime.now(timezone.utc).isoformat()} active flow runs: {active}") if active == 0: break if time.time() - started > TIMEOUT_SECONDS: print("Timed out waiting for drain", file=sys.stderr) sys.exit(1) time.sleep(POLL_SECONDS) for unit in WORKER_UNITS: run(["sudo", "systemctl", "stop", unit]) if name == "__main__": import anyio anyio.run(main) ``` --- My strongest recommendation for your setup Given what you described, I’d aim for this end state: 1. Consolidate to one worker per pool where possible 2. Use a templated systemd unit 3. Use a wrapper script with staged retry/backoff 4. For maintenance: - graceful planned outage: pause schedules, wait for drain, stop worker service - hard outage imminent: stop worker service immediately 5. After maintenance: - start worker service - resume schedules --- Useful Prefect docs / entry points - Prefect Cloud app: Prefect Cloud - Prefect docs home: Docs - Deployments concepts: Docs - Work pools / workers docs: Docs The docs links above are general because I wasn’t given page-specific URLs by the tools, and I don’t want to invent deep links. If you want, next I can turn this into a more opinionated production-ready bundle with: - one final

prefect-worker@.service

- one

run-prefect-worker.sh

- one

maintenance-down.sh

- one

maintenance-up.sh

- one

drain-and-stop.sh

tailored to either: - a single shared pool, or - multiple pools / multiple environments