Hi everyone, Is there any documentation available for hosting prefect open source in azure container...
k
Hi everyone, Is there any documentation available for hosting prefect open source in azure container apps?
n
@Marvin can you research how to host OSS prefect on azure container apps? should we add any docs on this?
m
thought for 487.4 seconds
n
generally speaking hosting OSS prefect is • hosting a machine that runs
prefect server start
• hosting a postgres database • hosting a redis instance • hosting some client machines that run
prefect worker start
the canonical way to do this at scale (on azure) is AKS, but you can certainly host the components yourself on serverless if that's what you want
m
Short answer: there’s no official Prefect integration or docs for Azure Container Apps (ACA) today. Prefect 3.x does have first‑class support for Azure Container Instances (ACI) and Kubernetes (AKS via Helm), and those are the recommended Azure paths. If you really want to run OSS Prefect on ACA, it’s doable, but you’ll be stitching things together yourself and should expect some rough edges. What’s officially supported (recommended) - Self-hosted Prefect Server - Docker Compose and K8s/Helm are documented and maintained - Self-hosted guide - Server Helm chart (works great on AKS) - Execution on Azure - Azure Container Instances (ACI) worker and push work pools are supported - Work pools concept: Work pools - Docker image: prefecthq/prefect:3-latest If you must use Azure Container Apps (ACA) Status: not officially supported or documented. Here’s a pragmatic outline that works, but is “use at your own risk”: - Prereqs - External Postgres (e.g., Azure Database for PostgreSQL Flexible Server) - External Redis (e.g., Azure Cache for Redis) - An ACA Environment (consider VNet integration if you need private DB/Redis) - Avoid scale-to-zero on server components (set min replicas = 1) - Split Prefect server into two ACA apps 1) API App - Image:
prefecthq/prefect:3-latest
- Command:
Copy code
prefect server start --host 0.0.0.0 --port 4200 --no-services
- HTTP ingress on port 4200 (public or private per your needs) - Configure env/secret settings for Postgres/Redis per the self-hosted guide above 2) Services App - Image:
prefecthq/prefect:3-latest
- Command:
Copy code
prefect server services start
- No ingress required; ensure it can reach Postgres and Redis - Notes: - Keep at least 1 replica for both, otherwise ACA scale-to-zero will stop the API/scheduler. - Add liveness/readiness probes and resource requests/limits. - Workers and flow run infrastructure - Strongly recommended: use Prefect’s Azure Container Instances integration for execution, even if the server runs on ACA. Create a work pool and use an ACI worker. - Create a push work pool (serverless):
Copy code
prefect work-pool create --type azure-container-instance:push --provision-infra my-aci-pool
- Or use a hybrid worker (polling) running anywhere you like (VM, container app, AKS node, etc.):
Copy code
prefect work-pool create --type azure-container-instance my-aci-pool
      prefect worker start --pool my-aci-pool --type azure-container-instance
- If you insist on a worker inside ACA, run another ACA app with the worker process:
Copy code
prefect worker start --pool <your-pool> --type azure-container-instance
Keep min replicas > 0 to avoid cold starts and missed polls. - Networking and settings - Make sure the worker can reach the Prefect API URL and Azure APIs for provisioning ACI jobs. - Store secrets (DB URL, Redis, Azure creds) as ACA secrets and map to env vars (see self-hosted guide for Prefect server settings). - Consider disabling Dapr sidecars unless you need them; they add overhead. Tradeoffs and caveats on ACA - Not tested/supported by Prefect; you’ll own the operational burden. - Scale-to-zero will break the server and/or worker; keep min replicas > 0. - Multi-container docker-compose patterns don’t translate 1:1; you’re managing multiple ACA apps and wiring them together. - Latency: ACA cold starts can add seconds to startup. Pre-warm server and worker replicas to mitigate. For execution, ACI push pools introduce provisioning latency per run; consider concurrency and job TTL tweaks to balance cost vs. latency. Should we add docs? - I’d recommend:
- A clear “Not officially supported” note on the self-hosted page that points Azure users to AKS (Helm) or ACI (workers/push pools). - Optionally, a short “community recipe” that outlines the ACA approach above, with a big disclaimer. It would save users time while making the support stance clear. - Proposed doc additions: - On Self-hosted, add a callout: “Azure Container Apps is not officially supported; use AKS (Helm) for production or ACI for execution.” - A small recipe page (Community) titled “Running Prefect Server on Azure Container Apps (unsupported)” with: - Architecture diagram (API app + Services app + external Postgres/Redis) - Commands shown above - Configuration checklist (secrets, ingress, min replicas, networking) - Link to ACI worker docs and work pool creation examples If you want, I can draft the “unsupported ACA recipe” page copy so we can decide whether to publish it in the community section. Would you plan to run just the Prefect server on ACA and keep execution on ACI, or put the worker in ACA as well?