You can indeed self-host Prefect on Kubernetes (like AWS EKS) without needing a Prefect Cloud account. This allows you to manage your own Prefect server and run flow executions independently. Here’s a bit of guidance on setting up and configuring Prefect for your needs:
1.
Self-Hosting Prefect:
- Use the
Prefect Helm chart to deploy a Prefect server on your Kubernetes cluster.
- You will have access to a Prefect UI and the API server, enabling you to manage flows similar to Prefect Cloud but without features like RBAC and SSO.
2.
Work Pools and Workers:
- A
work pool in Prefect allows structuring and managing the execution of flows by coordinating deployments with workers. Think of them as adaptive managers for your execution environments or infrastructure. More about this is warmly detailed
here.
-
Workers are essential in OSS (open-source) versions, and they act as agents executing flows. When deploying on Kubernetes, the workers would interact with available resources there to run your tasks.
- You can manage work pools and adjust them using the CLI commands such as
prefect work-pool ls
as detailed
here.
By tailoring these components according to your infrastructure needs, you can comfortably orchestrate your workflows. Remember, setting up the infrastructure might seem a maze, but with the right chart (or Helm), you'll navigate through smoothly! Feel free to explore more nuanced configurations directly in the documentation I linked above.
