Here’s the quickest way to upgrade Prefect installed via pip: 1) Activate the environment where Prefect is installed (virtualenv/conda/etc.) 2) Upgrade Prefect

python -m pip install -U prefect

- If you use extras (e.g., AWS, GCP, Kubernetes), include them:

python -m pip install -U "prefect[aws,gcp,kubernetes]"

- If you want to stay on Prefect 3.x explicitly:

python -m pip install -U "prefect>=3,<4"

3) Verify the installed version

prefect version
# or
python -c "import prefect; print(prefect.__version__)"

Tips: - If you’re not in a virtual environment and need a user install, add

--user

. - Restart any running workers after upgrading so they use the new version. Useful links: - Install/upgrade guide: Prefect 3.x installation docs - Release notes: Prefect 3.x release notes - CLI reference for version: prefect version command - Coming from Prefect 2.x? See the 3.0 notes and migration highlights: Version 3.0 release notes If you can share your environment (OS, venv/conda, any extras you use), I can tailor the exact command.

Short answer: yes, you can self-host Prefect and run Docker workers for that scale — but whether it stays “simple” depends on how many of those 50–70 flows run concurrently and your HA/auto-scaling needs. What works well with Docker workers - You run Prefect Server (OSS) yourself and one or more Docker worker hosts that pull work from a Docker work pool. - You control concurrency with worker limits and deployment/work-queue concurrency limits. - You’re okay scaling by adding more worker processes/hosts and managing VM capacity yourself. When to prefer Kubernetes - You expect 50–70 flows running concurrently, need auto-scaling, high availability, and robust scheduling/restarts. - You want easy resource requests/limits per run and better bin-packing at scale. Recommended self-hosted (Docker) plan 1) Stand up Prefect Server with Docker Compose - Required: PostgreSQL with the pg_trgm extension; Prefect Server. - Optional but recommended for larger installs: Redis for event messaging and Prefect background services. - Docs: Self-hosted with Docker Compose and Server concepts 2) Point your clients to your server

prefect config set PREFECT_API_URL=http://<your-server>:4200/api

3) Create a Docker work pool and run multiple workers - Create the pool:

prefect work-pool create "docker-pool" --type docker

- Start workers with per-process concurrency limits (run multiple instances across one or more hosts). For example, 7 workers with

--limit 10

each to allow ~70 concurrent runs:

prefect worker start --pool docker-pool --limit 10

Docs: Workers and Manage work pools 4) Create deployments that target the pool - Via code (Prefect 3.x pattern):

from prefect import flow

@flow
def my_flow():
    ...

# Deploy from a repo
my_flow.from_source(
    "<https://github.com/owner/repo>",
    entrypoint="path/to/file.py:my_flow",
).deploy(
    name="prod",
    work_pool_name="docker-pool",
    image="<http://ghcr.io/owner/image:tag|ghcr.io/owner/image:tag>",
)

- Or use a prefect.yaml and

prefect deploy

. Docs: Create deployments Operational tips for 50–70 long-running flows - Concurrency control - Use

--limit

on workers to cap concurrent runs per host. - Use deployment/work-queue concurrency limits to protect shared resources. - Capacity planning - Distribute workers across multiple hosts to avoid exhausting a single Docker daemon/VM. - Server dependencies - PostgreSQL with pg_trgm is required; size it to your expected run volume and enable backups. - Redis is optional; consider it for larger/multi-instance setups to improve event handling. - Reliability - Run background services for scheduling and housekeeping tasks. - Restart workers after version upgrades. Helpful docs - Self-hosted with Docker Compose: Guide - Server concepts: Concepts - Workers overview: Workers - Work pools: Manage work pools - Deployments: Create deployments A few clarifying questions to tailor the recommendation - Do you expect 50–70 flows to be running concurrently, or is that the total number of distinct flows with lower concurrency? - What are the typical CPU/RAM needs per flow run? - Do you require HA for the Prefect API/DB and zero-downtime upgrades? - Do you already run Kubernetes? If so, Kubernetes workers are likely the best long-term fit at this scale.

If your concurrency is moderate and you’re comfortable managing a handful of worker VMs, Docker workers with a self-hosted Prefect Server is a solid, simple starting point. If you truly need 50–70 concurrent long-running flows with elasticity and HA, I’d recommend Kubernetes workers.

If your flows are “always-on” loops and you don’t want to manage platform plumbing, the most sustainable path is to offload the control plane and keep your compute simple. Recommendation (in order of least ops to most) 1) Prefect Cloud + Push Work Pools (no workers to run) - What you manage: just your cloud account (ECS Fargate, Cloud Run, or Azure ACI). Prefect submits runs directly; no worker processes or servers to maintain. - Best when you want to avoid running workers entirely. - Suitability for long-running loops: ECS Fargate supports long tasks; still, we recommend chunking work into scheduled runs instead of one infinite run. - Docs: Serverless (Push) work pools 2) Prefect Cloud + Docker workers (minimal infra you own) - What you manage: one or a few small VMs with Docker and a worker process supervised by systemd; no Prefect server/DB to run. - Run multiple workers with

--limit

to cap concurrency; scale VM size/worker count as needed. - Good balance if you must stick with Docker but want low ops. - Start a worker:

prefect work-pool create "docker-pool" --type docker
prefect worker start --pool docker-pool --limit 10

- Docs: Workers | Manage work pools 3) Managed Kubernetes + Prefect Cloud + K8s workers (for scale/HA) - What you manage: a managed K8s cluster (EKS/GKE/AKS). Prefect schedules pods per run; autoscaling and HA are handled by the cluster. - Best if you truly have many concurrent long-running workflows or need elasticity. - Docs: Kubernetes workers Why Prefect Cloud? - Eliminates running/patching a Prefect server and database; you focus on your code and images. - Same developer experience for deployments and observability. - Concepts: Work pools overview Patterns for long-running “loop” work - Prefer scheduled, resilient runs over a single infinite loop (easier upgrades, retries, and observability). For example, schedule every minute and process a batch of work. - If you must loop inside a run, use async sleeps and heartbeats so runs stay healthy and don’t hog CPU:

from prefect import flow, get_run_logger
import asyncio

@flow
async def worker_loop():
    log = get_run_logger()
    while True:
        try:
            # do a unit of work
            <http://log.info|log.info>("processing...")
            # ... your logic ...
        except Exception as exc:
            log.exception(f"work unit failed: {exc}")
            await asyncio.sleep(5)  # short backoff on error
        else:
            await asyncio.sleep(30)  # don’t spin; yield CPU

- Add deployment-level concurrency limits to protect shared systems. - Use Automations to detect “zombie” runs and alert/restart if heartbeats stop: Detect zombie flows - Deployment basics and job variables: Create deployments | Job variables Putting it all together with minimal ops - Easiest: use Prefect Cloud + ECS Fargate push pool; you don’t run workers or a Prefect server. Deploy with

flow.deploy(..., work_pool_name="<your-ecs-push-pool>", image="...")

. Docs: Serverless (Push) work pools and AWS specifics: ECS guide - If you prefer Docker hosts: use Prefect Cloud + 1–3 Docker workers under systemd with

--limit

, and let Prefect handle orchestration. Quick clarifiers to tailor this for you - Are you open to using Prefect Cloud to avoid running the server/DB?

- Which cloud do you use (AWS/GCP/Azure)? Any constraints on using ECS/Cloud Run/ACI? - Roughly how many flows will be running concurrently and what are their CPU/RAM needs? With your answers, I can give you a concrete deploy plan and sizing guidelines.

Got it — if Prefect Cloud’s deployment constraints don’t work for you, you can run a simple, durable self‑hosted control plane and keep your compute on Docker workers with very little platform overhead. A minimal, sustainable setup - Control plane VM (self-hosted Prefect Server) - Run Prefect Server + Postgres with Docker Compose. - Required: PostgreSQL with pg_trgm extension; Prefect Server. - Optional (recommended as you scale): Redis and Prefect background services. - Docs: Self-host with Docker Compose and Server concepts - Worker VMs (your runtime) - 1–3 small VMs running Docker workers that poll a Docker work pool. - Run each worker under systemd so it auto-restarts and you don’t babysit it. High‑level steps 1) Bring up Prefect Server (control plane) - Follow the Compose guide: Guide - Make sure your Postgres has pg_trgm enabled. - Expose the API (default UI/API on :4200). 2) Point your clients/workers at your server

prefect config set PREFECT_API_URL=http://<server-host>:4200/api

3) Create a Docker work pool and start workers

prefect work-pool create "docker-pool" --type docker
prefect worker start --pool docker-pool --limit 10

- Run multiple workers (and/or on multiple hosts) to reach your target concurrency. - Docs: Workers | Manage work pools 4) Deploy flows and keep them running - Prefer scheduled runs that perform a “unit of work” each time, rather than a single infinite loop. This is easier to upgrade and recover. - Example (Prefect 3.x):

from prefect import flow

@flow
def do_work():
    # process one batch of work here
    ...

# Deploy from a repository
do_work.from_source(
    "<https://github.com/owner/repo>",
    entrypoint="path/to/file.py:do_work",
).deploy(
    name="prod",
    work_pool_name="docker-pool",
    image="<http://ghcr.io/your/image:tag|ghcr.io/your/image:tag>",
    interval=60,  # every 60 seconds
)

- If you must loop inside a single run, add sleeps/backoff so you don’t spin:

from prefect import flow, get_run_logger
import asyncio

@flow
async def worker_loop():
    log = get_run_logger()
    while True:
        try:
            # do a unit of work
            <http://log.info|log.info>("processing...")
        except Exception as exc:
            log.exception(f"work unit failed: {exc}")
            await asyncio.sleep(5)   # short backoff on error
        else:
            await asyncio.sleep(30)  # yield CPU; don’t tight-loop

Helpful references - Self-host with Docker Compose: Guide - Server concepts and settings: Concepts - Create deployments in 3.x: Create deployments - Customize job variables (env, mounts, etc.): Job variables - Detect stuck (“zombie”) flows: Zombie detection Operations minimalism tips - Put the control plane (Server + Postgres) on a single VM with Docker Compose; back up the Postgres volume daily. - Run workers under systemd so they auto-restart on failures or reboots. Example unit:

[Unit]
Description=Prefect Docker worker
After=network.target

[Service]
Environment=PREFECT_API_URL=http://<server-host>:4200/api
ExecStart=/usr/bin/env bash -lc 'prefect worker start --pool docker-pool --limit 10'
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

- Scale by adding more worker processes or hosts; use

--limit

and deployment/work-queue concurrency limits to protect shared systems. If you share:

- your approximate concurrent flow count and per-run CPU/RAM, - whether you want Redis in the first pass, I can provide a ready-to-run compose file and suggested VM sizes.

Here’s a safe, repeatable way to upgrade a self-hosted Prefect (Docker Compose) setup and what to expect during the change. What changes during an upgrade - Short control-plane downtime: the Prefect API/UI and scheduling will be unavailable while containers restart and database migrations run. - Running flow runs: continue on your Docker/K8s infrastructure; however, logs/state updates sent during the outage may be delayed or dropped. - New runs: won’t be scheduled/started during downtime. - Automations: won’t execute while the server is down. - Database: schema may migrate; always take a backup first. Pre-upgrade checklist - Read release notes for the target version and note any breaking changes: Prefect 3.x release notes - Back up Postgres (or the volume) used by Prefect Server. - Confirm your Compose file’s env vars still match current docs: Self-host with Docker Compose - Pick a maintenance window (usually a few minutes). Recommended procedure (maintenance window) 1) Quiesce new work - Option A: Pause the work pool(s) so no new runs are pulled

prefect work-pool pause "docker-pool"

- Option B: Pause schedules on critical deployments (optional if you pause pools). 2) Pull new images

docker compose pull

3) Stop server/services

docker compose stop
# or to recreate containers cleanly:
docker compose down

4) Upgrade the database schema - Run migrations in a one-off server container that has the same env (DB URL, etc.). Replace the service name with yours (often

prefect-server

):

docker compose run --rm prefect-server prefect server database upgrade

Docs for the database CLI: Server CLI 5) Start everything back up

docker compose up -d

6) Resume work

prefect work-pool resume "docker-pool"

Upgrading workers - If your workers run from a Prefect image (e.g.,

prefecthq/prefect:3-latest

): - Update the image tag if you pin versions, then recreate/restart worker containers. - If your workers use a system Python install: - Upgrade the package and restart the service:

python -m pip install -U "prefect>=3,<4"
systemctl restart prefect-worker   # if you use systemd

- Verify worker connectivity with:

prefect worker start --help

Expected impact - Downtime duration: typically seconds to a few minutes, depending on image pulls and DB migrations. - Running flows: keep executing; some logs and heartbeats may be missing during the outage. Once the API returns, state updates resume. - Scheduling/automations: paused during downtime; queued schedules resume after the server is back. Rollback plan - If something goes wrong: - Stop services:

docker compose down

- Restore your Postgres backup (or volume) - Start the previous images (pin the prior tags) and

docker compose up -d

- The DB CLI also supports downgrades if you know the target revision:

docker compose run --rm prefect-server prefect server database downgrade

Tips - Test the upgrade in a staging environment first using a copy of your DB. - Keep OSS server and clients/workers on the same major (3.x). Aim to minimize minor/version drift to avoid schema/API mismatches. - If you use Redis or background services in Compose, include them in `pull`/`up -d` so they match the server version. Useful links - Docker Compose self-host guide: Guide - Server CLI (start, database upgrade/downgrade): Server CLI - Work pools and workers: Workers | Manage work pools - Release notes: Prefect 3.x release notes

If you share your current Compose services (names and images), I can give you exact

docker compose

commands and any environment tweaks to watch for.