k

    kiran

    1 year ago
    Hi y’all. I’m getting started with Prefect this week and very excited! Question about script logic: does the
    with Flow(...):
    context manager basically replace using a
    main()
    function? i.e., instead of something like this:
    from prefect import task, Flow
    
    @task
    def say_hello():
        print("Hello, world!")
    
    @task
    def say_goodbye():
        print("Goodbye, world!")    
    
    
    with Flow("My First Flow") as flow:
        say_hello()
        say_goodbye()
    Would
    main()
    be used with the context manager (and is it weird/redundant to do so)? i.e., would you decorate each function with
    @task
    , order them in
    main()
    (but not decorate
    main()
    ) and then call
    main()
    from the context manager like so:
    from prefect import task, Flow
    
    
    @task
    def say_hello():
        print("Hello, world!")
    
    
    @task
    def say_goodbye():
        print("Goodbye, world!")
    
    
    def main():
        say_hello()
        say_goodbye()
    
    
    if __name__ == "__main__":
        with Flow("My First Flow") as flow:
            main()
    Thanks for any advice/insights on best practices around this!
    d

    DK

    1 year ago
    I think it's unnecessary to use if name == "main" in general with prefect flows. The context manager is kind of like a main function. It's the highest level where your functions(tasks) are called. But I sometimes do use a main() function to group tasks together. For example, if you have 100 tasks, and you had 10 flows that are exactly the same except for their parameters, you could use a the same main function under each Flow, instead of calling each task under each Flow.
    emre

    emre

    1 year ago
    Welcome @kiran! I'm sure there are other approaches within the community, but I'm gonna talk about what I usually do, and why I do those things. 1- Wrap the context manager with a
    build_flow()
    function. 2- Use
    main()
    to configure my
    build_flow()
    call, and also customize what to do with the built flow.
    def build_flow(is_polite):
        with Flow('abc') as f:
            say_hello()
            if is_polite:
                say_goodbye()
        return flow
        
    def main():
        # Here I can customize the flow I build without modifying the context manager
        flow = build_flow(True)
    
        # What I wanna do with the flow resides in main separated from what the flow consists of.
        # flow.run()
        flow.visualize()
    Usually I implement a simple CLI inside my
    main()
    function, so I can pick what I wanna do and build without touching anything in the code.
    davzucky

    davzucky

    1 year ago
    I concur with @emre we create it own command line tool that standardize all the customization we need. Good to have the drain of the flow in one function as well
    k

    kiran

    1 year ago
    Thanks all for your input! Good things for me to think about. @emre are you not using Cloud to run your flows?
    emre

    emre

    1 year ago
    Not yet, but if i did, I would add registration as a cli option, then cloud would handle prod runs, and flow.run would handle development
    k

    kiran

    1 year ago
    Ah great. Thanks!