Hey folks, is it possible when running a script with prefect, to specify which flow to run? E.g. im...

Federico Zambelli

04/21/2023, 10:00 AM

Hey folks, is it possible when running a script with prefect, to specify which flow to run? E.g. imagine this:

Copy code

@flow
def first_flow():
    ...

@flow
def second_flow():
    ...

def main():
    first_flow()
    second_flow()

What if in certain situations, i want to run only the 2nd flow? Is that possible / reasonable ?

Deceivious

04/21/2023, 10:05 AM

Copy code

if <cond1>:
    first_flow()
if <cond2>:
    second_flow()

Federico Zambelli

04/21/2023, 10:06 AM

yeah this I imagined was the poor man's solution. I was wondering if prefect had something built-in for this

Deceivious

04/21/2023, 10:08 AM

I would maybe rename main and make it a flow and trigger the other 2 flows as sub flows if you only want a single deployment. But cant say more without more context on why that would be

a poor man's solution

. Its simple and it works.

Federico Zambelli

04/21/2023, 10:09 AM

yeh sorry, poor man's solution probably was not the proper term. Lemme illustrate better

Federico Zambelli

04/21/2023, 10:12 AM

Imagine you have a task (or a bunch of) that return unpickleable objects, and therefore can't be cached. This is my current case, as i have a couple of tasks that return async generators. Eventually in the first flow I persist the result of these generators to disk (first flow), and later do something else with this results, reading from disk (2nd flow) I would like to, say, skip the execution of first flow if input parameters are the same as the last ones used, for instance.

Deceivious

04/21/2023, 11:16 AM

Your first task could be

Copy code

def generate_or_load_items(): #just an example
    if file.exists():
       return load_items()
    return generate_items()

But yes prefect doesnt have in built support for this use case as far as I am aware.

Federico Zambelli

04/21/2023, 11:17 AM

alright fair enough, thanks for the help!

🙌 1

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.