Thanks for flagging this — when a single-container Prefect Server is under load, the bottlenecks are almost always the database and API concurrency. Here’s a practical path to speed up the UI and reduce resource usage in a self-hosted setup (assuming Prefect 3.x):
High‑impact, quick wins
- Move off a single container to the supported Docker Compose stack so the API/UI and Postgres run as separate services you can size independently. See
Prefect Server via Docker Compose.
- Put Postgres on fast storage with enough memory (at least 2–4 vCPU and 8–16 GB RAM for moderate load) and give it its own persistent volume.
- Upgrade to the latest 3.x release; we regularly ship performance improvements.
- In the UI, reduce the default time range when browsing runs (large ranges query lots of rows).
Database health and retention (most important for UI speed)
- Prefect’s UI performance is directly tied to database size/health, especially flow runs, task runs, logs, artifacts, and events.
- Implement routine retention and maintenance:
- Delete or archive old runs/logs/artifacts on a schedule.
- Run VACUUM/ANALYZE (or pg_repack for large bloat) regularly after deletes.
- Consider connection pooling with PgBouncer.
- We provide guidance and example approaches here:
Database maintenance for Prefect Server.
- If you’re writing lots of logs/artifacts, reduce volume where possible (e.g., log at INFO/WARN instead of DEBUG, avoid large artifacts for every run).
Scale the API/UI
- Horizontal scale: run multiple API replicas behind a reverse proxy/load balancer (e.g., NGINX). This smooths out spikes in UI/API traffic.
- Reverse proxy: add NGINX to handle keep-alives, gzip compression, and caching of static UI assets. Example reference:
Docker Compose with reverse proxy example (community).
- If you’re staying on Docker, give the API container dedicated CPU/memory limits/requests. If you outgrow Compose, prefer Kubernetes.
Tune Prefect Server settings
- Prefect exposes settings for:
- Database connection pooling, statement timeouts
- Enabling/disabling background services (e.g., schedulers)
- Logging volume and logging-to-API
- Exact names vary by version; configure them via environment variables per our settings reference:
Prefect 3.x settings reference.
- Start by:
- Increasing the DB connection pool size if you see frequent connection churn
- Lengthening statement timeouts only if you expect long-running queries
- Disabling any background services you do not need
- Reducing logging-to-API if you don’t need logs in the UI for every run
Recommended production topology
- Docker Compose (good for small/medium loads)
- Postgres as a dedicated service with tuned parameters (shared_buffers, work_mem, autovacuum settings)
- Prefect API/UI as separate services; scale API replicas if needed
- Optional PgBouncer in front of Postgres
- Optional NGINX as a reverse proxy in front of API/UI
- Docs:
Docker Compose guide
- Kubernetes (best for HA and growth)
- Use the official Helm chart to deploy the Server and scale API replicas
- Docs:
Prefect Server Helm and Helm repo
prefect-helm
Observability
- Watch Postgres metrics (connections, locks, bloat, slow queries) and API latency.
- If the DB is consistently >60–70% CPU or I/O bound, scale it up or move to a managed Postgres.
If you share a bit more detail, I can give you a targeted plan:
- Approx. number of flow/task runs per day and log volume
- Current DB size and Postgres resources (vCPU/RAM/IOPS)