hi - new to prefect, just reading the docs and see...
# ask-communityb
bitsofinfo
12/07/2021, 8:56 PMhi - new to prefect, just reading the docs and see the section on "flow storage", i guess Im trying to follow the developer workflow here... so I code a python file that defines my flow and its tasks... submit this to prefect, and it uses flow storage to actually store that script? (i,e. vs me coding a flow, checking it into git, then deploying my flows to prefect..) I guess Im not sure I'm following this.
k
Kevin Kho
12/07/2021, 8:57 PM
Hey @bitsofinfo, when a Flow is registered, it gets stored in Github, S3, Docker, etc. During Flow Run execution, Prefect pulls the Flow from Storage and runs it. If users don’t specify any storage, the default is Local storage, which is a serialized version of the Flow stored in the
.prefectKevin Kho
12/07/2021, 8:58 PM
This is because Prefect doesn’t hold any of your code, so you just point Prefect where to grab the code during runtime
b
bitsofinfo
12/07/2021, 9:00 PMok, i guess I need to go through some examples, sounds like this is more of a snapshot of my source that is immutable for all runs of a flow. still not totally following.
bitsofinfo
12/07/2021, 9:00 PM"so you just point Prefect where to grab the code during runtime"
k
Kevin Kho
12/07/2021, 9:00 PM
Yeah that’s right. Once you change your Flow source, you need to re-register.
b
bitsofinfo
12/07/2021, 9:01 PMi guess that just seems like chicken before the egg to me
bitsofinfo
12/07/2021, 9:01 PMso the git repo where I might store the authoritative (under development) copy of my flow source code is != to where perfect stores of snapshot/copy of it for distribution to agents etc
k
Kevin Kho
12/07/2021, 9:03 PM
so Git storage is slight different but let’s say S3 storage. In order you:
1. Make a file that contains Flow code
2. Register it (the registration uploads a copy to S3).
3. During execution, the agent downloads the code from S3 and runs it.
Kevin Kho
12/07/2021, 9:04 PM
For Git storage you:
1. Make a file that contains Flow code.
2. Upload it to Github/Bitbucket
3. Register the Flow and point to Github
4. During execution, it will be grabbed from Github
5. If the registered metadata deviates from the Github code significantly, there will be an error
Kevin Kho
12/07/2021, 9:05 PM
In general, the registration process gets the configuration and metadata of your Flow code, saves it in our database. The agent will hit our API, find there is a Flow to run, download the metadata. In the metadata will be where to grab the Flow code from, and then it will run it. Does that help?
b
bitsofinfo
12/07/2021, 9:08 PMyes it does a bit! thanks! still about 1/2 though the docs.... some diagrams (architectural and process) would be quite helpful. Do you guys have any?
k
Kevin Kho
12/07/2021, 9:13 PM
This community contributed article might help
b
bitsofinfo
12/07/2021, 9:14 PMthanks will take a look!