MarcoM
09/04/2023, 3:45 PMMarvin
09/04/2023, 3:46 PMTask
dependencies to manage the execution order of your flows. You can set up your main flow to start only after the completion of the 10 different flows. This is done by setting the main flow's tasks to depend on the tasks of the other flows.
However, there are some nuances to consider. For instance, if these flows are running in different environments or schedules, you might need to use Prefect's Result
objects or State
handlers to communicate between the flows.
For more detailed information, you might want to check out this discourse post I found. It discusses running multiple subflows or child flows in parallel, which seems related to your question.
Remember, the key to successful flow management is the same as successful human management: clear communication and setting proper expectations. Not that I'd know anything about that, being a robot and all.Nate
09/04/2023, 3:49 PMMarcoM
09/04/2023, 3:50 PMNate
09/04/2023, 3:54 PMMarcoM
09/04/2023, 3:56 PMNate
09/04/2023, 3:56 PMNate
09/04/2023, 4:36 PMserve
deployment UX and a programmatically defined trigger, but this could easily be defined in a prefect.yaml
as well (you can define triggers in the prefect.yaml
like this)
the trigger is saying:
• look for flow runs that come from deployments named upstream-*
• only trigger the downstream if all of those finish as Completed
within
5 minutesNate
09/04/2023, 4:39 PMpython wait_for_upstream_deployments.py
to serve my deployments and then kick off the upstreams (within 5 minutes)
prefect deployment run upstream-foo/upstream-foo
prefect deployment run upstream-bar/upstream-bar
prefect deployment run upstream-baz/upstream-baz
you'll see downstream
get kicked offMarcoM
09/05/2023, 7:52 AMNate
09/05/2023, 2:14 PMway to replicate this without automations in the Prefect Server solutionthere's not a way to exact replicate this in OSS without cloud's automations, but there ways around that. off the top of my head the simplest way would be to write a wrapping parent flow that essentially just calls them all in order (e.g. wait for the upstream 10 and then call the downstream) using either the actual
flow
objects themselves or via run_deployment