Hey, I want to create a dynamic DAG (where flows are not directly written into the code using the @flow decorator) However, I need to set dependencies for these subflows so a proper DAG is created. But Prefect 2.0 doesn't seem to have any argument for setting the upstream dependencies for flows. What could be a possible workaround for this?
✅ 1
a
Anna Geller
08/22/2022, 10:46 AM
this is not strictly needed since flows are blocking; in example below, Prefect will wait for the staging_area subflow before moving on to the data_mart subflow
Copy code
@flow(description="Fail immediately if any subflow fails")
def data_warehouse_load():
staging_area()
data_mart()
data_services()
m
Malavika S Menon
08/22/2022, 10:49 AM
I was trying for a case where the flows are not defined strictly in order inside the main flow. Flows are called using a loop/recursion. In such a case, how do I set a logical dependency between the flows? I guess I am looking for something equivalent to the wait_for argument in tasks
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.