Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
t
Trip Humphrey
09/05/2021, 9:45 PMHowdy everyone! I'm Trip, and I work as Healthcare Data Consultant in Virginia. I have about 15 years experience in Healthcare, IT, and Data. Currently I am evaluating a number of solutions that pertain to data workload orchestration. I am curious about prefect, but I will admit I am having a bit of trouble parsing the seemingly nuance differences between Flow Storage and the concept of version control of the Flow .py files themselves. Do we use a traditional git repo with cicd workflows providing change control of the actual code? I know I can store the flows in a number of different ways, but I just can't seem to get it to click as to how a team might collaboratively manage updates to those flows over time.
👋 11
z
Zanie
09/05/2021, 11:24 PMHi! https://github.com/PrefectHQ/prefect/discussions/4042 may be a little more in depth than you're looking for, but I'd recommend taking a look.
But basically, I'd recommend using git storage and registering the flows in CI/CD to point at the commit. This way git is your source of truth for versioning and you don't need to copy a pickled flow to another location (ie
S3Storaget
Trip Humphrey
09/05/2021, 11:31 PMCould I keep my git repo as the source files and use regular cicd to push them to storage as docker images as well? I will take a look at the link you sent. Is the level of redundancy I'm suggesting superfluous or are there advantages to splitting the development process from the flow storage medium?
z
Zanie
09/05/2021, 11:32 PMYep definitely, people do that as well.
The advantages are entirely dependent on your team / setup, it's hard to say. Generally, unless you need your flow's execution environment to be locked at registration time and specified per flow,
DockerStorageDockerRunGithubStorageYou can also do things like register flows into different projects for your main or dev branches while using git storage (to be clear, you can do this with docker storage too, it's just a lot of images to be building)
t
Trip Humphrey
09/05/2021, 11:36 PMWell I really appreciate the fast response and warm welcome to the community. I will read through the link you shared and let you know how it ends up meshing with the workflow I have in my head.