Howdy again, I am trying to wrap my head around how to deliver <MLOps Level 1> using Prefect for the...
j
Joe Still
11/01/2021, 5:31 PMHowdy again, I am trying to wrap my head around how to deliver MLOps Level 1 using Prefect for the automated pipeline. Can anyone point me to a resource that addresses dev/stage/prod pipeline deployment with Prefect? For Airflow I believe I can treat it like a common application deployment building my DAGs into a docker image I deploy at each stage. Maybe I just don’t understand how Prefect flows are managed in this way or am thinking incorrectly about pipeline promotion through dev/stage/prod. Thoughts?
Joe Still
11/01/2021, 5:33 PMI see how to use Prefect in a single stage environment, you just register your flows no problem. But what common patterns exist to package and deploy (and revert) Prefect flows between dev/stage/prod.
Joe Still
11/01/2021, 5:34 PMHere is what I was looking at for Airflow to simply deliver my pipelines with docker, though I still think Prefect will better serve us if I can understand automated pipeline deployment between stages.
Joe Still
11/01/2021, 5:39 PMAlso trying to understand how this would work in a multi-tenant solution. With Airflow I had been considering a workload per DAG group per dev/stage/prod based on the built docker image. For context, I am looking into a centralized orchestration solution for multiple teams and have not ruled out Cloud though am currently thinking in terms of self-hosted.
k
Kevin Kho
11/01/2021, 5:55 PM
Hi @Joe Still, a couple of things here. First is that multi-tenancy is a Cloud only enterprise feature. Server only has 1 tenant.
That said, most of the users only on one tenant use Projects as the logical separation for dev, staging, and prod. They also have dedicated hardware by specifically labelling their agents so that there is a separation between environments.
For the CI/CD portion, I think you can start with this thread where a bunch of users have outlined their solutions.
j
Joe Still
11/01/2021, 5:57 PMThe single-tenant / multi-project concept is very helpful ty and ty for the link, looks like exactly what I wanted to read tyty
k
Kevin Kho
11/01/2021, 5:59 PM
Feel free to chime in there as well!
🙌 1
Prefect Community
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by