Newskooler

    Newskooler

    1 year ago
    Hi Prefect community ! 👋 Could someone (or more) please share any best practices in regards to registering a flow? If I have my code in a repository and then this repository deployed to a server where prefect is running (server and agent) what is an elegant way of registering a flow? currently i need to open some kind of python (ipython or notebook) and register manually each flow (and flow version). Surely there must be a more elegant way?: )
    Robin

    Robin

    1 year ago
    I second that. We are also currently registering flows manually, but I would like to set up a deploy stage using GitLab CICD to register flows. I am also looking forward to reading from others who have already successfully done this 🙂
    Newskooler

    Newskooler

    1 year ago
    Hi @Robin If you were to set up a deploy stage, how would that work?
    Robin

    Robin

    1 year ago
    If it works out as I would like it, the flow would be reregistered whenever pushing to a (specific) branch on GitLab.
    Does that answer your question or are you interested in the technical details?:
    Newskooler

    Newskooler

    1 year ago
    No, I don’t quite understand technically how I would approach it.
    for example: do you wrap the
    flow
    generation into a function, which you call somehow? What is your flow version has not changed (do you still register); how is the actual registration done, etc.
    Robin

    Robin

    1 year ago
    Ah, OK! I was just thinking of running the
    flow.py
    file on GitLab using python whenever all previous stages have succeeded (i.e. all tests successful).
    Newskooler

    Newskooler

    1 year ago
    so your
    flow.py
    contain
    flow.register()
    ?
    I personally think this may help deployment: https://github.com/PrefectHQ/prefect/issues/3433
    Robin

    Robin

    1 year ago
    Hmm, it's true that gitlab would reregister all the flows whenever a new commit is pushed to the respective branch (and all tests succeed), which might be unwanted... Question being why this would be bad?
    Newskooler

    Newskooler

    1 year ago
    It’s not gitlab that does it; it’s
    flow.register()
    which does it (just to clarify in case of confusion). I think there is no need to have different versions which are consecutive and identical. It’s just clutter. Separately (and neatly) it does help with deployment because in case anyone deploys by running
    flow.register()
    (say a cronjob or CI or whatever) then it will only register different flows.
    Robin

    Robin

    1 year ago
    Yeah, that's understood. I agree that unnecessary clutter should be avoided and that the functionality that you suggest makes sense.
    Newskooler

    Newskooler

    1 year ago
    If that’s the case, feel free to support on the Issues (no pressure : D )
    Robin

    Robin

    1 year ago
    😄 Has not such a high priority for us atm. Good luck with it!
    Jacob Blanco

    Jacob Blanco

    1 year ago
    We are working on a blog post on this particular topic. We make us of gitpython to detect changes in flow files, load the flow objects from each file, and then run
    flow.register()
    Each flow is stored in a separate folder with a configuration YAML that lets people define a bunch of options, like the flow project name, additional Python dependencies, etc…..
    Newskooler

    Newskooler

    1 year ago
    Pleaae share here when ready. Would love to read it.
    j

    jorwoods

    1 year ago
    @Jacob Blanco Is that blog post available?
    Jacob Blanco

    Jacob Blanco

    1 year ago
    Not yet, other priorities came up. Let me see if we have something rough that I can share with you.