<@ULVA73B9P> My single docker instance of Prefect ...
# ask-marvin
k
@Marvin My single docker instance of Prefect Server is struggling with high resource utilization and slow UI. How can I solve this in my self hosted environment?
m
thought for 308.9 seconds
Thanks for flagging this — when a single-container Prefect Server is under load, the bottlenecks are almost always the database and API concurrency. Here’s a practical path to speed up the UI and reduce resource usage in a self-hosted setup (assuming Prefect 3.x): High‑impact, quick wins - Move off a single container to the supported Docker Compose stack so the API/UI and Postgres run as separate services you can size independently. See Prefect Server via Docker Compose. - Put Postgres on fast storage with enough memory (at least 2–4 vCPU and 8–16 GB RAM for moderate load) and give it its own persistent volume. - Upgrade to the latest 3.x release; we regularly ship performance improvements. - In the UI, reduce the default time range when browsing runs (large ranges query lots of rows). Database health and retention (most important for UI speed) - Prefect’s UI performance is directly tied to database size/health, especially flow runs, task runs, logs, artifacts, and events. - Implement routine retention and maintenance: - Delete or archive old runs/logs/artifacts on a schedule. - Run VACUUM/ANALYZE (or pg_repack for large bloat) regularly after deletes. - Consider connection pooling with PgBouncer. - We provide guidance and example approaches here: Database maintenance for Prefect Server. - If you’re writing lots of logs/artifacts, reduce volume where possible (e.g., log at INFO/WARN instead of DEBUG, avoid large artifacts for every run). Scale the API/UI - Horizontal scale: run multiple API replicas behind a reverse proxy/load balancer (e.g., NGINX). This smooths out spikes in UI/API traffic. - Reverse proxy: add NGINX to handle keep-alives, gzip compression, and caching of static UI assets. Example reference: Docker Compose with reverse proxy example (community). - If you’re staying on Docker, give the API container dedicated CPU/memory limits/requests. If you outgrow Compose, prefer Kubernetes. Tune Prefect Server settings - Prefect exposes settings for: - Database connection pooling, statement timeouts - Enabling/disabling background services (e.g., schedulers) - Logging volume and logging-to-API - Exact names vary by version; configure them via environment variables per our settings reference: Prefect 3.x settings reference. - Start by: - Increasing the DB connection pool size if you see frequent connection churn - Lengthening statement timeouts only if you expect long-running queries - Disabling any background services you do not need - Reducing logging-to-API if you don’t need logs in the UI for every run Recommended production topology - Docker Compose (good for small/medium loads) - Postgres as a dedicated service with tuned parameters (shared_buffers, work_mem, autovacuum settings) - Prefect API/UI as separate services; scale API replicas if needed - Optional PgBouncer in front of Postgres - Optional NGINX as a reverse proxy in front of API/UI - Docs: Docker Compose guide - Kubernetes (best for HA and growth) - Use the official Helm chart to deploy the Server and scale API replicas - Docs: Prefect Server Helm and Helm repo prefect-helm Observability - Watch Postgres metrics (connections, locks, bloat, slow queries) and API latency. - If the DB is consistently >60–70% CPU or I/O bound, scale it up or move to a managed Postgres. If you share a bit more detail, I can give you a targeted plan: - Approx. number of flow/task runs per day and log volume - Current DB size and Postgres resources (vCPU/RAM/IOPS)
- Prefect version, and whether you’re on a single container or Compose already - Whether you can run Kubernetes, or you prefer to stay on Docker Helpful links - Docker Compose reference: Self-hosted with Docker Compose - Advanced self-hosting patterns/HA: Advanced self-hosted - Settings reference (env vars): Settings reference - Database maintenance and cleanup: Database maintenance - Helm chart for Kubernetes: Server Helm and repo prefect-helm