https://prefect.io logo
Title
a

aaron

02/24/2022, 6:00 PM
hey community, I have a question that I've been unable to find an answer for in the docs. say I have two mutually exclusive flows, Flow A and Flow B. Flow B should not be executed until Flow A has completed successfully. how can I declare a dependency on Flow A from within Flow B? dagster has something similar to this and I was hoping that prefect does as well.
k

Kevin Kho

02/24/2022, 6:03 PM
Hey @aaron, I think the paradigm here would be to use the
create_flow_run
task at the end of Flow A to trigger Flow B. You can also do it on the Flow B side, but it becomes harder because you have to poll for state using the GraphQL API. It will also take some thought how to point it to a specific Flow run of Flow A You can also have an outer main flow that is responsible for calling
create_flow_run
for both Flow A and Flow B.
a

aaron

02/24/2022, 6:09 PM
thanks for the quick response! so there is no way to explicitly declare a dependency on an upstream task or flow by name, just triggering downstream? I realize my example just used two flows, and while your recommended approach would work for a few downstream dependents but if I have dozens of pipelines that are dependent on Flow A's successful execution, triggering them all from Flow A would get cumbersome.
k

Kevin Kho

02/24/2022, 6:12 PM
But let’s say Flow B is checking if it can start, and Flow A has two copies, how does B know which copy of A it is dependent on?
Or are you only guaranteed one concurrent run of Flow A?
I read their docs, are you talking about Graphs to stitch together multiple jobs?
a

aaron

02/24/2022, 6:21 PM
yes, I believe so
k

Kevin Kho

02/24/2022, 6:23 PM
I think the equivalent is a Flow of Flows in Prefect because they use the Graph to stitch together jobs and run them together. For us, the outer Graph would just be a “main” flow where you have all your flow logic. This is the docs page for that
🙌 1
a

aaron

02/24/2022, 6:23 PM
and yes, just one concurrent run of Flow A. let's say this Flow refreshes a data source once daily. Other flows depend on this data source being up to date, but they should not run until Flow A has completed. Is the only way to accomplish this currently in Prefect to query the graph?
k

Kevin Kho

02/24/2022, 6:24 PM
I think the flow of flows paradigm would be much easier. But yes we can also do it from the GraphQL API, but I think that’s fighting the framework honestly
a

aaron

02/24/2022, 6:27 PM
yeah, I would have to agree. thanks for your help.
k

Kevin Kho

02/24/2022, 6:28 PM
I am still reading about this Graph. The Graph definition seems pretty analogous to the main flow except they are more concerned with inputs and outputs. I think for you, you should just be aware that you can do the same with Parameters:
with Flow(...) as flow:
    x = Paramater("x", 1)
    flow_a = create_flow_run(...,parameters={"x": x})
    flow_b = create_flow_run(...,parameters={"x": x})