Here’s a clear, 3.x-focused overview of Prefect Server and its role in the architecture. Assumption: You’re using Prefect 3.x. If you’re on 2.x, let me know and I can tailor this. What is Prefect Server - Prefect Server is the self-hosted orchestration backend for Prefect. It provides the API and UI that manage flow/task state, scheduling, concurrency limits, events/automations, logs, artifacts, blocks/secrets, deployments, work pools/queues, and more. - It offers the same core orchestration engine as Prefect Cloud; the main difference is you run and operate it yourself. Where it fits in the architecture - Your flow code runs on workers in your infrastructure (local, Docker, Kubernetes, etc.). - Workers poll Prefect Server for work, execute flows, then report state/logs/artifacts back to the Server. - The Server stores metadata and drives orchestration logic; it does not execute your code. Core responsibilities of Prefect Server - API and UI - Serves the REST API used by the SDK, workers, and UI - Hosts the Prefect UI for managing and monitoring runs - Orchestration - Validates and records flow/task run state transitions - Applies orchestration rules and retries - Scheduling - Creates scheduled flow runs from your deployments’ schedules - Maintains “min future runs” so queued work exists for workers to pick up - Work distribution - Maintains work pools and queues - Lets workers poll for the next eligible runs - Concurrency limits - Enforces global and tag-based concurrency to protect shared resources - Events and automations - Ingests events from flows and external systems - Triggers automations (e.g., run a deployment, pause/cancel, send a webhook) - Data persistence - Stores runs, deployments, logs, artifacts metadata, blocks/secrets (encrypted), variables, events - Uses SQLite by default for dev; PostgreSQL recommended for production - Observability - Retains logs and run history - Streams logs/events to the UI How workers/executors interact with Server - Workers poll your work pool/queue for the next runs to execute. - When they pick up work, they set the run state to pending/running, stream logs, write task states, and finally set completed/failed, etc. - The Server records all of this and enforces scheduling/concurrency rules, but does not run your code. What Prefect Server does NOT do - It does not run your flow code. Execution happens on workers. - It does not store your flow code. Deployments reference your code location (e.g., Git repo, storage block). - It does not push work to workers (workers pull). - It does not manage your compute infrastructure. - It does not expose enterprise features like SSO, advanced RBAC, or managed SLAs (those are in Cloud). Key components inside Server - API server (FastAPI): All endpoints for flows/runs, deployments, work pools/queues, logs, artifacts, blocks, variables, events/automations, concurrency limits - Background services: - Scheduler: materializes future scheduled runs - Event/automation processor: matches events to triggers and executes actions - Foreman/health processes: track worker heartbeats, mark pools ready, cleanup tasks - Database layer: Stores all orchestration data (SQLite or Postgres) - Optional messaging/Redis: Used for high-scale event ordering, concurrency leasing, and streaming How it differs from Prefect Cloud (brief) - Same API and core orchestration logic - Server is self-hosted; you manage the database, scaling, uptime, security hardening - Cloud adds enterprise features (RBAC, SSO, audit logs), managed uptime, and operational simplicity Good links to go deeper - Concepts: Prefect Server overview: docs: Server - Self-hosting (CLI, Docker Compose, Helm, scaling): - docs: Server CLI - docs: Docker Compose

- docs: Helm chart - docs: Scaling self-hosted - Workers and work pools: docs: Workers and docs: Work pools - Deployments: docs: Deployments - Events and automations: docs: Automations - Blocks and secrets: docs: Blocks - Concurrency limits: docs: Concurrency limits - Source code: GitHub: Prefect If you want, tell me your environment (Docker/Kubernetes/VMs, database choice, expected scale), and I can suggest a right-sized self-hosted Server setup.