<@ULVA73B9P> are there any documented best practic...
# ask-marvin
m
@Marvin are there any documented best practices on code organization and integration with CI/CD? We're moving from Prefect on-premises to Prefect Cloud, and would like to take the opportunity to clean up our architecture while we do that. I've looked through most of the documentation on the website, but haven't come away with a conclusive answer yet, and I'm hoping you can summarize/aggregate the available info to fit my requirements. Summary: I will be introducing Prefect to people who don't have any prior experience with it. I'd like to remove as much friction as possible from their experience by doing the following: • Organizing all of our Prefect workflows in a mono-repo, instead of separate repos per workflow. ◦ This means I'd lean towards packaging the flow code into the Docker container during the build step, rather than pulling from a Git repo • Using code-based deployments instead of prefect.yaml files wherever possible • Creating CI/CD scripts that automatically deploy any workflows that have changed since the last PR merge • Setting up 3 environments: local (on each dev's machine, using Prefect OSS installed locally in Docker as their back-end), dev (a Prefect cloud environment), and Prod (another Prefect cloud environment). Can you give me any tips on best practices to accomplish this? We'll be running a hybrid architecture, where things will be built in the CI/CD pipeline by an agent that is hosted on the same server as we'll run flows on. So we don't need to worry about pushing Docker repos up to a registry, they should be available locally.
n
hey @Mike B - sorry marvin went awol here this template repo (https://github.com/zzstoatzz/prefect-pack) might be helpful in some respects
@Marvin hello
m
Hello. How can I assist you today?
n
@Marvin are there any documented best practices on code organization and integration with CI/CD? We're moving from Prefect on-premises to Prefect Cloud, and would like to take the opportunity to clean up our architecture while we do that. I've looked through most of the documentation on the website, but haven't come away with a conclusive answer yet, and I'm hoping you can summarize/aggregate the available info to fit my requirements. Summary: I will be introducing Prefect to people who don't have any prior experience with it. I'd like to remove as much friction as possible from their experience by doing the following: • Organizing all of our Prefect workflows in a mono-repo, instead of separate repos per workflow. ◦ This means I'd lean towards packaging the flow code into the Docker container during the build step, rather than pulling from a Git repo • Using code-based deployments instead of prefect.yaml files wherever possible • Creating CI/CD scripts that automatically deploy any workflows that have changed since the last PR merge • Setting up 3 environments: local (on each dev's machine, using Prefect OSS installed locally in Docker as their back-end), dev (a Prefect cloud environment), and Prod (another Prefect cloud environment). Can you give me any tips on best practices to accomplish this? We'll be running a hybrid architecture, where things will be built in the CI/CD pipeline by an agent that is hosted on the same server as we'll run flows on. So we don't need to worry about pushing Docker repos up to a registry, they should be available locally.
🧐 I'll have to look into why marvin is balking here
Using code-based deployments instead of prefect.yaml files wherever possible
this example (https://github.com/zzstoatzz/prefect-pack/tree/main/flows/with_special_deps) is cloning the code, but switching to baking code in the image is just removing
from_source
and updating the dockerfile to copy the code
Setting up 3 environments: local (on each dev's machine, using Prefect OSS installed locally in Docker as their back-end), dev (a Prefect cloud environment), and Prod (another Prefect cloud environment).
use profiles
prefect profile --help
: users can have
local
dev
and
prod
each with their own PREFECT_API_URL etc
Creating CI/CD scripts that automatically deploy any workflows that have changed since the last PR merge
https://github.com/zzstoatzz/prefect-monorepo/blob/main/.github/workflows/env-separated-deploy.yml
m
Great, thanks for your help @Nate!