https://prefect.io logo
t

Tobias Schmidt

09/28/2019, 1:57 PM
I feel like this must be pretty basic: Is there a way to combine/string together multiple flows?
z

Zachary Hughes

09/29/2019, 7:10 PM
Hi Tobias, not at the moment. If you want to express dependencies, your best bet is to do so using tasks within a flow. You can also call
flow.run()
in a task, but I'm not sure if that's what you're looking for.
d

Dylan

09/30/2019, 11:16 AM
Hi Tobias, can you tell me a little more about your use case? In what way are your flows dependent on one another?
t

Tobias Schmidt

09/30/2019, 1:34 PM
Not in any sort of deep way. I had thought of a flow as a distinct sequence of steps and had thought that it’d be easy to compose these sequences such that one must run after another
l

Luke Orland

09/30/2019, 2:32 PM
would either of these options work?
Untitled
c

Chris White

09/30/2019, 3:03 PM
interesting - the
set_upstream
pattern with
reference_tasks
that @Luke Orland is suggesting, coupled with
flow.update(other_flow)
might do the trick: https://docs.prefect.io/api/unreleased/core/flow.html#prefect-core-flow-flow-update
m

Mark Koob

09/30/2019, 6:28 PM
I've been able to do this successfully (at least for a PoC) using
.update
and
.replace
. The result is correct but not entirely satisfying since routing the results correctly requires some a priori knowledge of the task names. It looks like
set_upstream
may be a better approach?
t

Tobias Schmidt

10/01/2019, 8:05 AM
@Luke Orland Yeah, both of those would work. setting an entire flow as upstream would mean that the entire flow has concluded vs. just requiring that its reference tasks have run to completion?
More generally, it seems to me like it’d be nice for me to be able to treat a flow like a task. That is, to be able to compose a flow not of tasks but of sub-flows that are logically distinct units
m

Mark Koob

10/01/2019, 2:06 PM
It appears that making a
class FlowTask(prefect.Task)
where the run method is running another flow as @Zachary Hughes suggests works reasonably well. I'm not sure if this hinders optimizations, though.
(I happen to be working on a similar problem which is why I'm stalking this thread)
2 Views