https://prefect.io logo
n

Newskooler

10/05/2020, 10:24 AM
Hi Prefect community ! 👋 Could someone (or more) please share any best practices in regards to registering a flow? If I have my code in a repository and then this repository deployed to a server where prefect is running (server and agent) what is an elegant way of registering a flow? currently i need to open some kind of python (ipython or notebook) and register manually each flow (and flow version). Surely there must be a more elegant way?: )
r

Robin

10/05/2020, 11:55 AM
I second that. We are also currently registering flows manually, but I would like to set up a deploy stage using GitLab CICD to register flows. I am also looking forward to reading from others who have already successfully done this 🙂
👍 2
n

Newskooler

10/05/2020, 11:55 AM
Hi @Robin If you were to set up a deploy stage, how would that work?
r

Robin

10/05/2020, 12:06 PM
If it works out as I would like it, the flow would be reregistered whenever pushing to a (specific) branch on GitLab.
Does that answer your question or are you interested in the technical details?:
n

Newskooler

10/05/2020, 12:07 PM
No, I don’t quite understand technically how I would approach it.
for example: do you wrap the
flow
generation into a function, which you call somehow? What is your flow version has not changed (do you still register); how is the actual registration done, etc.
r

Robin

10/05/2020, 12:40 PM
Ah, OK! I was just thinking of running the
flow.py
file on GitLab using python whenever all previous stages have succeeded (i.e. all tests successful).
n

Newskooler

10/05/2020, 12:41 PM
so your
flow.py
contain
flow.register()
?
👍 1
I personally think this may help deployment: https://github.com/PrefectHQ/prefect/issues/3433
r

Robin

10/05/2020, 12:59 PM
Hmm, it's true that gitlab would reregister all the flows whenever a new commit is pushed to the respective branch (and all tests succeed), which might be unwanted... Question being why this would be bad?
n

Newskooler

10/05/2020, 1:03 PM
It’s not gitlab that does it; it’s
flow.register()
which does it (just to clarify in case of confusion). I think there is no need to have different versions which are consecutive and identical. It’s just clutter. Separately (and neatly) it does help with deployment because in case anyone deploys by running
flow.register()
(say a cronjob or CI or whatever) then it will only register different flows.
r

Robin

10/05/2020, 1:09 PM
Yeah, that's understood. I agree that unnecessary clutter should be avoided and that the functionality that you suggest makes sense.
👍 1
n

Newskooler

10/05/2020, 1:10 PM
If that’s the case, feel free to support on the Issues (no pressure : D )
r

Robin

10/05/2020, 1:28 PM
😄 Has not such a high priority for us atm. Good luck with it!
👍 1
j

Jacob Blanco

10/05/2020, 11:05 PM
We are working on a blog post on this particular topic. We make us of gitpython to detect changes in flow files, load the flow objects from each file, and then run
flow.register()
Each flow is stored in a separate folder with a configuration YAML that lets people define a bunch of options, like the flow project name, additional Python dependencies, etc…..
😍 1
👍 2
n

Newskooler

10/06/2020, 4:53 PM
Pleaae share here when ready. Would love to read it.
j

jorwoods

02/25/2021, 2:47 PM
@Jacob Blanco Is that blog post available?
j

Jacob Blanco

02/26/2021, 1:10 AM
Not yet, other priorities came up. Let me see if we have something rough that I can share with you.
🙏 1