Hey, I want to create a dynamic DAG (where flows a...
# prefect-community
Hey, I want to create a dynamic DAG (where flows are not directly written into the code using the @flow decorator) However, I need to set dependencies for these subflows so a proper DAG is created. But Prefect 2.0 doesn't seem to have any argument for setting the upstream dependencies for flows. What could be a possible workaround for this?
this is not strictly needed since flows are blocking; in example below, Prefect will wait for the staging_area subflow before moving on to the data_mart subflow
Copy code
@flow(description="Fail immediately if any subflow fails")
def data_warehouse_load():
I was trying for a case where the flows are not defined strictly in order inside the main flow. Flows are called using a loop/recursion. In such a case, how do I set a logical dependency between the flows? I guess I am looking for something equivalent to the wait_for argument in tasks
I checked and it looks like doing
is on the roadmap - this will return a future and will allow you to handle state dependencies as you like until then, this might be a workaround https://discourse.prefect.io/t/how-to-create-a-flow-run-from-deployment-orchestrator-pattern/803
so yes as you said, this way you will be able to pass wait_for list to a subflow the same way you can do today with tasks
Okay thanks
btw here is how you can pass data dependencies between subflows https://discourse.prefect.io/t/how-to-pass-data-between-subflows/721 (just updated)
🙌 1