Short answer: you don’t have to move if serve is working for you on a persistent machine. But if you want higher availability, easier scaling, or environment isolation, moving to a work pool + worker is the recommended path in Prefect 3.x—especially since you’re already on Prefect Cloud. How to decide - Stay on serve if: - You have a small number of flows on a single, reliable VM/host - You don’t need per-flow environment isolation (same Python/env for all) - Occasional maintenance windows (restarts) are acceptable - Move to work pools if you want: - High availability (multiple workers can poll the same pool) - Horizontal scaling and easier ops (independent worker lifecycle) - Environment isolation (process, Docker, or Kubernetes per workload) - Governance features (pool/queue-level concurrency, templates) - Option to adopt serverless/managed/push pools later Key differences - serve: a single long‑running process on your host polls your deployments and runs them locally. If that process stops, runs won’t start until it’s back up. - work pools + workers: workers poll the pool and execute runs; you can run multiple workers for HA and choose infra (process, Docker, K8s, etc.). This decouples scheduling from execution and plays very well with Cloud. Links - Work pools (concepts) - Workers (concepts) Lightweight migration on persistent infra (process worker) 1) Create a process work pool and start a worker on your host:

prefect work-pool create "my-process-pool" --type process
prefect worker start --pool my-process-pool

2) Deploy your flow to the pool (Prefect 3.x pattern):

from prefect import flow

@flow
def my_flow():
    ...

# Create/update a deployment that targets your process pool
my_flow.deploy(
    name="my-deployment",
    work_pool_name="my-process-pool",
    cron="0 * * * *",  # or interval/rrule/schedules
)

3) Cut over from serve with no missed runs: - Keep your existing serve process running initially. - Start the worker (above) and confirm it’s picking up test runs from the new deployment in Cloud. - When you’re ready to stop serve, note that serve pauses schedules by default on shutdown. Either set

pause_on_shutdown=False

in your serve call during the transition, or unpause the old deployment in the UI if you need it to keep scheduling. Notes - In Prefect 3.x, avoid the removed patterns like

prefect deployment build

or

Deployment.build_from_flow()

. Use

flow.deploy(...)

(or

flow.from_source(...).deploy(...)

) instead. - If you want stronger isolation on the same host, consider a Docker work pool and run a Docker worker; if you’re in K8s, use a Kubernetes pool/worker. If you share roughly how many flows you run, whether you need per-flow environments, and how critical HA is, I can give a firmer recommendation and a tailored migration plan.

Short answer: moving to a work pool won’t magically eliminate API polling—workers also poll the API—but it can make this much easier to control and observe, which helps prevent “ghost process” explosions that trigger Cloud rate limits. What likely happened - Each “ghost” serve process polls the API every ~10s. Ten stray processes = ~60 calls/min each ≈ 600/min, plus heartbeats/logs/state updates from runs. That can trip rate limits quickly. How work pools help - Centralize and observe: You’ll see each worker in Cloud (with last heartbeat). If duplicates appear, it’s obvious and easy to stop them. - One poller per worker: Put all 15 deployments in one pool and run 1–2 workers. That’s 1–2 pollers instead of many serve processes. - Tunable polling: Workers expose clear settings to slow polling and prefetch windows. - Operational control: Use systemd/containers to ensure exactly one worker instance and named workers per host. If you want to stay on serve - You can still reduce API calls by slowing the runner’s poll interval: - Set (example) environment variable:

export PREFECT_RUNNER_POLL_FREQUENCY=30

- Make sure there is only one serve process per host. Manage it with systemd and a PID/lockfile to prevent duplicates. - But serve gives you less visibility into duplicates compared to workers. If you move to a work pool on the same host (recommended at your scale: ~15 deployments/30 flows) 1) Create a process pool and start a single worker:

prefect work-pool create "prod-process-pool" --type process
prefect worker start --pool prod-process-pool --name host-01

2) Redeploy your flows to the pool (Prefect 3.x pattern):

from prefect import flow

@flow
def my_flow():
    ...

my_flow.deploy(
    name="my-deployment",
    work_pool_name="prod-process-pool",
    cron="0 * * * *",  # or your schedule
)

3) Slow worker polling to reduce API calls:

export PREFECT_WORKER_QUERY_SECONDS=30
export PREFECT_WORKER_PREFETCH_SECONDS=30
# optional: longer heartbeat if desired
export PREFECT_WORKER_HEARTBEAT_SECONDS=60

4) Cutover cleanly: - Keep your current serve process running until the worker is up and you’ve validated a test run. - Then stop serve. Remember serve pauses schedules on shutdown by default—set

pause_on_shutdown=False

temporarily during the transition or unpause as needed in the UI. Operational guardrails - Give each worker a unique name (

--name

) and manage it with systemd to ensure only one instance. - Use Cloud’s work pool/worker views to spot duplicates quickly. - Set a pool- or queue-level concurrency limit to bound activity if needed. Will this “fix” rate limits? - It addresses the root cause (unseen duplicate pollers). With one or two well-managed workers and slower polling, you’ll dramatically reduce API traffic. - If you keep serve and prevent duplicates and slow polling, that can also work—but workers make prevention and detection much easier. Docs - Work pools (concepts) - Workers (concepts) If you share how those ghost processes were created (systemd, Docker, manual shells), I can suggest exact systemd unit files or container run settings to harden things further.

✅ thought for 290.7 seconds

Great questions. 1) How many workers to start for ~15 deployments? - Start with 1 worker per pool, then scale up if you see backlogs or need HA. - One process worker can service many deployments; it polls the pool and will pick up any scheduled runs it has capacity for. - Control concurrency with: - Worker-level:

prefect worker start --pool <pool> --limit N

- Runner subprocess cap:

PREFECT_RUNNER_PROCESS_LIMIT

(default 5). Effective parallelism is min(worker --limit, runner process limit). - Server-side caps: pool/queue/deployment concurrency limits. - For your footprint (30 flows across 15 deployments on persistent infra), a single well-sized worker (limit 5–10) is usually enough. Add a second worker for HA once things are stable. Tip: Since your earlier issue was “ghost processes,” keep the worker count small and explicit. Run each worker under systemd/containers with a clear

--name

so duplicates are obvious in Cloud. 2) Is there a “define then activate” pattern like to_deployment() + serve? Yes. Keep your “define first” approach, just change how you activate: Option A: Define with to_deployment → register with apply (no local serving)

d = flow.to_deployment(
    name="my-deployment",
    work_pool_name="my-pool",
    cron="0 * * * *",
    tags=["prod"],
)
deployment_id = d.apply()  # registers deployment in Cloud
# Start/keep a worker running: prefect worker start --pool my-pool

- This is the closest analog to your current pattern, but runs via workers instead of a serve loop. - Schedules/triggers live on the deployment; the worker will pick them up. Option B: Use flow.deploy(...) directly (convenience)

flow.deploy(
    name="my-deployment",
    work_pool_name="my-pool",
    cron="0 * * * *",
)

- Equivalent outcome, just fewer lines. If you add an image, it will build/push as needed. Operational knobs to avoid API rate limits - Keep pollers few and intentional. For one worker: - Slow polling:

export PREFECT_WORKER_QUERY_SECONDS=30
    export PREFECT_WORKER_PREFETCH_SECONDS=30
    export PREFECT_WORKER_HEARTBEAT_SECONDS=60

- Right-size concurrency:

export PREFECT_RUNNER_PROCESS_LIMIT=5   # or 10
    prefect worker start --pool my-pool --limit 5

- Use server-side caps if helpful: - Pool:

prefect work-pool set-concurrency-limit <limit> <pool>

- Queue:

prefect work-queue set-concurrency-limit <limit> <queue>

- Deployment:

flow.deploy(..., concurrency_limit=N)

Docs for reference - Work pools (concepts) - Workers (concepts) If you share your typical parallelism needs (how many runs overlap at peak), I can suggest concrete values for

--limit

and

PREFECT_RUNNER_PROCESS_LIMIT

, and a sample systemd unit to keep exactly one worker alive.

Great clarifier — I meant parallelism at the flow-run level, not task-level. - Worker/runner parallelism = how many flow runs can execute at the same time on that machine - Controlled by: -

prefect worker start --pool <pool> --limit N

→ max flow runs this worker will submit concurrently -

PREFECT_RUNNER_PROCESS_LIMIT

→ max subprocesses actually running flow runs (default 5) - Effective per-worker concurrency = min(

--limit

,

PREFECT_RUNNER_PROCESS_LIMIT

) - This is independent of task parallelism inside a flow - Task-level parallelism = within a single flow run - Your

DaskTaskRunner

continues to handle this; nothing changes there - Just remember each concurrent flow run can also spin up/attach to a Dask cluster and consume resources; size

--limit

accordingly How to choose a value - Think “how many deployments might run at the same time at peak?” - If your HH:00 schedules can trigger, say, 5 flows at once and your host can handle 5 concurrent Dask-backed flows, start with:

export PREFECT_RUNNER_PROCESS_LIMIT=5
    prefect worker start --pool my-pool --limit 5

- If some flows are heavy, cap them with a deployment-level concurrency limit (so only 1 of that deployment runs at once) and keep a higher worker limit for the rest. - Excess scheduled runs just wait for capacity; they may show as Late but will start when a slot frees. Define-then-activate pattern (like to_deployment + serve) - Yes, use

to_deployment(..., work_pool_name=...)

then register with `apply()`:

d = flow.to_deployment(
    name="my-deployment",
    work_pool_name="my-pool",
    cron="0 * * * *",
    tags=["prod"],
)
deployment_id = d.apply()  # registers it in Cloud
# Keep a worker running: prefect worker start --pool my-pool [--limit N]

- Or the convenience method:

flow.deploy(name="my-deployment", work_pool_name="my-pool", cron="0 * * * *")

Optional: mirror your current “logical groups” - If those 5 serve calls were just for organization, you can create 5 work queues within one pool and either: - Run a single worker that pulls all queues, or - Run one worker per queue with

--work-queue <name>

for clearer separation Docs - Work pools (concepts) - Workers (concepts) If you share your peak “simultaneous flow runs” and how heavy each is with Dask, I can suggest concrete values for

--limit

and

PREFECT_RUNNER_PROCESS_LIMIT

(and, if helpful, per-deployment concurrency limits).