Howdy again, I am trying to wrap my head around how to deliver MLOps Level 1 using Prefect for the automated pipeline. Can anyone point me to a resource that addresses dev/stage/prod pipeline deployment with Prefect? For Airflow I believe I can treat it like a common application deployment building my DAGs into a docker image I deploy at each stage. Maybe I just don’t understand how Prefect flows are managed in this way or am thinking incorrectly about pipeline promotion through dev/stage/prod. Thoughts?
I see how to use Prefect in a single stage environment, you just register your flows no problem. But what common patterns exist to package and deploy (and revert) Prefect flows between dev/stage/prod.
Also trying to understand how this would work in a multi-tenant solution. With Airflow I had been considering a workload per DAG group per dev/stage/prod based on the built docker image. For context, I am looking into a centralized orchestration solution for multiple teams and have not ruled out Cloud though am currently thinking in terms of self-hosted.
10 months ago
Hi @Joe Still, a couple of things here. First is that multi-tenancy is a Cloud only enterprise feature. Server only has 1 tenant.That said, most of the users only on one tenant use Projects as the logical separation for dev, staging, and prod. They also have dedicated hardware by specifically labelling their agents so that there is a separation between environments.For the CI/CD portion, I think you can start with this thread where a bunch of users have outlined their solutions.
10 months ago
The single-tenant / multi-project concept is very helpful ty and ty for the link, looks like exactly what I wanted to read tyty