Howdy everyone! I'm Trip, and I work as Healthcare...
# introductionst
Trip Humphrey
09/05/2021, 9:45 PMHowdy everyone! I'm Trip, and I work as Healthcare Data Consultant in Virginia. I have about 15 years experience in Healthcare, IT, and Data. Currently I am evaluating a number of solutions that pertain to data workload orchestration. I am curious about prefect, but I will admit I am having a bit of trouble parsing the seemingly nuance differences between Flow Storage and the concept of version control of the Flow .py files themselves. Do we use a traditional git repo with cicd workflows providing change control of the actual code? I know I can store the flows in a number of different ways, but I just can't seem to get it to click as to how a team might collaboratively manage updates to those flows over time.
👋 11
z
Zanie
09/05/2021, 11:24 PM
Hi! https://github.com/PrefectHQ/prefect/discussions/4042 may be a little more in depth than you're looking for, but I'd recommend taking a look.
Zanie
09/05/2021, 11:25 PM
But basically, I'd recommend using git storage and registering the flows in CI/CD to point at the commit. This way git is your source of truth for versioning and you don't need to copy a pickled flow to another location (ie
S3Storaget
Trip Humphrey
09/05/2021, 11:31 PMCould I keep my git repo as the source files and use regular cicd to push them to storage as docker images as well? I will take a look at the link you sent. Is the level of redundancy I'm suggesting superfluous or are there advantages to splitting the development process from the flow storage medium?
z
Zanie
09/05/2021, 11:32 PM
Yep definitely, people do that as well.
Zanie
09/05/2021, 11:34 PM
The advantages are entirely dependent on your team / setup, it's hard to say. Generally, unless you need your flow's execution environment to be locked at registration time and specified per flow,
DockerStorageDockerRunGithubStorageZanie
09/05/2021, 11:35 PM
You can also do things like register flows into different projects for your main or dev branches while using git storage (to be clear, you can do this with docker storage too, it's just a lot of images to be building)
t
Trip Humphrey
09/05/2021, 11:36 PMWell I really appreciate the fast response and warm welcome to the community. I will read through the link you shared and let you know how it ends up meshing with the workflow I have in my head.
4 Views