https://prefect.io logo
#introductions
Title
# introductions
t

Trip Humphrey

09/05/2021, 9:45 PM
Howdy everyone! I'm Trip, and I work as Healthcare Data Consultant in Virginia. I have about 15 years experience in Healthcare, IT, and Data. Currently I am evaluating a number of solutions that pertain to data workload orchestration. I am curious about prefect, but I will admit I am having a bit of trouble parsing the seemingly nuance differences between Flow Storage and the concept of version control of the Flow .py files themselves. Do we use a traditional git repo with cicd workflows providing change control of the actual code? I know I can store the flows in a number of different ways, but I just can't seem to get it to click as to how a team might collaboratively manage updates to those flows over time.
👋 11
z

Zanie

09/05/2021, 11:24 PM
Hi! https://github.com/PrefectHQ/prefect/discussions/4042 may be a little more in depth than you're looking for, but I'd recommend taking a look.
But basically, I'd recommend using git storage and registering the flows in CI/CD to point at the commit. This way git is your source of truth for versioning and you don't need to copy a pickled flow to another location (ie
S3Storage
)
t

Trip Humphrey

09/05/2021, 11:31 PM
Could I keep my git repo as the source files and use regular cicd to push them to storage as docker images as well? I will take a look at the link you sent. Is the level of redundancy I'm suggesting superfluous or are there advantages to splitting the development process from the flow storage medium?
z

Zanie

09/05/2021, 11:32 PM
Yep definitely, people do that as well.
The advantages are entirely dependent on your team / setup, it's hard to say. Generally, unless you need your flow's execution environment to be locked at registration time and specified per flow,
DockerStorage
is overkill and you can instead do a
DockerRun
with a base image of your own then
GithubStorage
to pull your flow into the container at runtime.
You can also do things like register flows into different projects for your main or dev branches while using git storage (to be clear, you can do this with docker storage too, it's just a lot of images to be building)
t

Trip Humphrey

09/05/2021, 11:36 PM
Well I really appreciate the fast response and warm welcome to the community. I will read through the link you shared and let you know how it ends up meshing with the workflow I have in my head.
4 Views