I’m trying to keep a large project organized and I’m wondering if I’m thinking about it the wrong way. I have two series of tasks currently defined as flows that are defined in seperate .py files. I was hoping to create a third flow that executed the first two and then performed a few more tasks. Is it better to think of those two upstream files as flows, or should I just organize each of those files by having them end in a final task that has all the upstream tasks as “upstream tasks”, and then import the terminal task into the master flow? (hopefully thats clear)

Hey @saml, I was following until the last sentence. I think the way to think about this is as subflows that you invoke with the

StartFlowRun

task and

create_flow_run

task in the third flow and then put your additional tasks after that. The important of the terminal task into the master flow won’t work I think

just so we’re speaking the same language: flow_a: a-1, a2, etc flow_b: b1, b2, etc flow_c: flow_a, flow_b, c1, c2, etc

StartFlowRun

will run a registered flow so the flow_a and flow_b need to be registered in the backend (Cloud or Server) for this to work.

ok that helps, so I can’t run it locally from a single file like I would when testing one of my example flows (flow_a)?

For the

StartFlowRun

task, no you can’t. I have an idea that I’ll mention but I’m not sure it’ll work. Let me show you

@task
def abc(...):
    return 1

@task
def bcd(...):
    return 2

with Flow(...) as flow:
    abc()

# notice flow is already defined. we can add another task

with flow:
    bcd()

so in that example, I would only define the “with flow” clause on the master flow and then import tasks from the other files and add them to the context?

To keep appending to a flow, you’d need to like, define it in

flow A

, and then

flow B

imports

flow

and adds more tasks and then

flow C

imports the updated

flow

and then adds more tasks. So it’s still pretty ugly

yeah that seems like it would be hard to get the order right.

Or you can not have flow objects in

flow_a

and

flow_b

. Just import that tasks from

flow_c

and construct the whole flow there. Really though I guess the standard way is creating subflows. The experience is rough for sure so for Prefect 2.0 these subflows will be a first class part of the main flow.

ok, maybe I’ll try that.

Oh wow if that works, that’s great! Have not seen it myself but as long as the visualization is good, I think this would work

wow so apparently this works:

from prefect import task, Flow
from time import sleep

from case_flow import flow as case_flow



@task
def test_final_task(other_task_str:str):
    print("final task sleep")
    print(other_task_str)
    sleep(10)
    print("done sleeping")

@task
def other_task():
    return "I'm another task"

with Flow("other flow") as other_flow:
    other_task()

final_flow = Flow("parent-flow")
final_flow.update(case_flow)
final_flow.add_task(test_final_task(other_task_str=other_flow.get_tasks(name='other_task'), flow=final_flow , upstream_tasks=[case_flow.reference_tasks()]))
final_flow.visualize()
final_flow.run()

Nicely done! Have not seen anyone do it like this.

Assuming I’m not missing something significant, this seems like it opens up a lot of options for flexible execution. Each sub flow could be easily run on it’s own if needed, or run from a parent flow without changing the code of the sub flow

So a lot of users get flexible execution from the

StartFlowRun

already since you can just register the subflow and then invoke it from the main flow. I think the flexibility this provides is on the development side for sure.

does startflowrun allow for the handing of objects from one flow or another though? I thought it just returned flow state, or could be set to wait for the flow to finish

Ah I see what you mean. That’s right. The answer is not in a first class way. In 0.15.0 there were 3 tasks introduced

create_flow_run

,

wait_for_flow_run

, and

get_task_run_result

. The

get_task_run_result

loads it from the result location. With your current set-up though, it will indeed be memory since they all will get registered together

interesting, I feel like i’m kind of getting away with something, which is usually a sign that I’m getting a little too creative for my own good, but hopefully this works ok