Joe Still

    Joe Still

    10 months ago
    Howdy again, I am trying to wrap my head around how to deliver MLOps Level 1 using Prefect for the automated pipeline. Can anyone point me to a resource that addresses dev/stage/prod pipeline deployment with Prefect? For Airflow I believe I can treat it like a common application deployment building my DAGs into a docker image I deploy at each stage. Maybe I just don’t understand how Prefect flows are managed in this way or am thinking incorrectly about pipeline promotion through dev/stage/prod. Thoughts?
    I see how to use Prefect in a single stage environment, you just register your flows no problem. But what common patterns exist to package and deploy (and revert) Prefect flows between dev/stage/prod.
    Here is what I was looking at for Airflow to simply deliver my pipelines with docker, though I still think Prefect will better serve us if I can understand automated pipeline deployment between stages.
    Also trying to understand how this would work in a multi-tenant solution. With Airflow I had been considering a workload per DAG group per dev/stage/prod based on the built docker image. For context, I am looking into a centralized orchestration solution for multiple teams and have not ruled out Cloud though am currently thinking in terms of self-hosted.
    Kevin Kho

    Kevin Kho

    10 months ago
    Hi @Joe Still, a couple of things here. First is that multi-tenancy is a Cloud only enterprise feature. Server only has 1 tenant. That said, most of the users only on one tenant use Projects as the logical separation for dev, staging, and prod. They also have dedicated hardware by specifically labelling their agents so that there is a separation between environments. For the CI/CD portion, I think you can start with this thread where a bunch of users have outlined their solutions.
    Joe Still

    Joe Still

    10 months ago
    The single-tenant / multi-project concept is very helpful ty and ty for the link, looks like exactly what I wanted to read tyty
    Kevin Kho

    Kevin Kho

    10 months ago
    Feel free to chime in there as well!