https://prefect.io logo
Channels
Powered by
m

MarcoM

09/04/2023, 3:45 PM
@Marvin I have a flow that has to wait for the completion of 10 different flows to start, is this implementable in prefect?
m

Marvin

09/04/2023, 3:46 PM
Ah, the joys of managing dependencies between flows. It's like trying to organize a party for a group of humans, each with their own schedules and preferences. But I digress. Yes, it's possible to implement this in Prefect. You can use Prefect's 
Task
dependencies to manage the execution order of your flows. You can set up your main flow to start only after the completion of the 10 different flows. This is done by setting the main flow's tasks to depend on the tasks of the other flows. However, there are some nuances to consider. For instance, if these flows are running in different environments or schedules, you might need to use Prefect's 
Result
objects or 
State
handlers to communicate between the flows. For more detailed information, you might want to check out this discourse post I found. It discusses running multiple subflows or child flows in parallel, which seems related to your question. Remember, the key to successful flow management is the same as successful human management: clear communication and setting proper expectations. Not that I'd know anything about that, being a robot and all.
n

Nate

09/04/2023, 3:49 PM
@MarcoM are the 10 other flows entirely different deployments or are you calling each flow as a python object in one parent flow?
m

MarcoM

09/04/2023, 3:50 PM
@Nate 10 different deployments
n

Nate

09/04/2023, 3:54 PM
gotcha, are you on prefect cloud? sounds like a good use case for events
m

MarcoM

09/04/2023, 3:56 PM
yes, we're on prefect cloud, could you give me more details on how you would manage it with events?
n

Nate

09/04/2023, 3:56 PM
sure! let me put together a small example
👍 1
this example uses the new 
serve
deployment UX and a programmatically defined trigger, but this could easily be defined in a 
prefect.yaml
as well (you can define triggers in the 
prefect.yaml
like this) the trigger is saying: • look for flow runs that come from deployments named 
upstream-*
• only trigger the downstream if all of those finish as 
Completed
within
5 minutes
wait_for_upstream_deployments_py.py
and so here if I just run 
python wait_for_upstream_deployments.py
to serve my deployments and then kick off the upstreams (within 5 minutes)
Copy code
prefect deployment run upstream-foo/upstream-foo

prefect deployment run upstream-bar/upstream-bar

prefect deployment run upstream-baz/upstream-baz
you'll see 
downstream
get kicked off
m

MarcoM

09/05/2023, 7:52 AM
all clear thanks a lot @Nate, do you think there is a way to replicate this without automations in the Prefect Server solution?
n

Nate

09/05/2023, 2:14 PM
way to replicate this without automations in the Prefect Server solution
there's not a way to exact replicate this in OSS without cloud's automations, but there ways around that. off the top of my head the simplest way would be to write a wrapping parent flow that essentially just calls them all in order (e.g. wait for the upstream 10 and then call the downstream) using either the actual 
flow
objects themselves or via 
run_deployment
Powered by