Thread
#prefect-community
    Scott Zelenka

    Scott Zelenka

    2 years ago
    Looking for best practices around continuous delivery around Flows. Specifically, we have a Flow that's triggered from another system over GraphQL. We're currently triggering the Flow on the
    version_group_id
    . The execution of the Flow includes a step to write data back to a Prod API of an external system. The challenge comes when making iterations to this Flow in development. The external system's Dev instance still triggers our Flow through GraphQL, but is expecting it to write data back to a Dev API. We could update and deploy the Flow to Cloud, but then the
    version_group_id
    picks up our Dev Flow, rather than our Prod Flow. The only thing I can think of is to have two different Flows deployed on Cloud (one for Prod and another for Dev). But in that case, promoting from Dev to Prod has a bunch of manual steps prone to human error. Interested in the communities thoughts on how you handle deploying multiple versions of the same Flow between Dev and Prod environments, where the configuration between Dev and Prod are different.
    j

    Joe Schmid

    2 years ago
    @Scott Zelenka what we've done is tack on environment names to our flows when we register from CICD, e.g. MyFlow-dev, MyFlow-staging, etc. Then we set labels to match so that an agent in the correctly environment picks up the flow. It feels like there should be a better long-term approach, but this is working ok for now.
    Scott Zelenka

    Scott Zelenka

    2 years ago
    Thanks. In our case we're agnostic to which Agent the Flow runs on. There's a difference in the execution of the Flow itself (WRT which API it writes back to) between Dev and Prod. I imagine we could have a parameter input to specify which lifecycle it's on .. but that doesn't help protect Prod from breaking changes during Dev iterations if it's all in the same Flow.
    j

    Joe Schmid

    2 years ago
    Yeah, it's interesting. Our CICD approach is still evolving but we want to have a typical progression something like this:1. Pull request to dev branch -> testing -> merge to dev & deploy Flow to dev 2. PR to staging branch -> testing -> merge to staging & deploy Flow to staging 3. PR to master branch -> testing -> merge to master & deploy Flow to prod
    nicholas

    nicholas

    2 years ago
    I think at the moment @Joe Schmid’s solution is the best approach to a CI pipeline I've seen. This is something we're actively looking at though and hope to have something more streamlined and robust soon
    Tyler Wanner

    Tyler Wanner

    2 years ago
    I understand your frustration with the inability to “promote”… In my mind, there’s a docker workflow where you’re building a versioned image off of the Prefect core image that contains the code to actually register the flow (rather than the flow itself), and then you promote that image to “prod”, which then kicks off a deploy to your -prod flow
    I believe this is similar to how Joe is doing things, but we are absolutely interested in any feedback and CI use-cases and I would love to put together something as early as this month around CI best practices. So, I’d love to help