< Marvin> I have a flow that has to wait for the completion Prefect Community #ask-marvin

<@ULVA73B9P> I have a flow that has to wait for th...

MarcoM

09/04/2023, 3:45 PM

@Marvin I have a flow that has to wait for the completion of 10 different flows to start, is this implementable in prefect?

Marvin

09/04/2023, 3:46 PM

Ah, the joys of managing dependencies between flows. It's like trying to organize a party for a group of humans, each with their own schedules and preferences. But I digress. Yes, it's possible to implement this in Prefect. You can use Prefect's

Task

dependencies to manage the execution order of your flows. You can set up your main flow to start only after the completion of the 10 different flows. This is done by setting the main flow's tasks to depend on the tasks of the other flows. However, there are some nuances to consider. For instance, if these flows are running in different environments or schedules, you might need to use Prefect's

Result

objects or

State

handlers to communicate between the flows. For more detailed information, you might want to check out this discourse post I found. It discusses running multiple subflows or child flows in parallel, which seems related to your question. Remember, the key to successful flow management is the same as successful human management: clear communication and setting proper expectations. Not that I'd know anything about that, being a robot and all.

Nate

09/04/2023, 3:49 PM

@MarcoM are the 10 other flows entirely different deployments or are you calling each flow as a python object in one parent flow?

MarcoM

09/04/2023, 3:50 PM

@Nate 10 different deployments

Nate

09/04/2023, 3:54 PM

gotcha, are you on prefect cloud? sounds like a good use case for events

MarcoM

09/04/2023, 3:56 PM

yes, we're on prefect cloud, could you give me more details on how you would manage it with events?

Nate

09/04/2023, 3:56 PM

sure! let me put together a small example

👍 1

Nate

09/04/2023, 4:36 PM

this example uses the new

serve

deployment UX and a programmatically defined trigger, but this could easily be defined in a

prefect.yaml

as well (you can define triggers in the

prefect.yaml

like this) the trigger is saying: • look for flow runs that come from deployments named

upstream-*

• only trigger the downstream if all of those finish as

Completed

within

5 minutes

wait_for_upstream_deployments_py.py

Nate

09/04/2023, 4:39 PM

and so here if I just run

python wait_for_upstream_deployments.py

to serve my deployments and then kick off the upstreams (within 5 minutes)

Copy code

prefect deployment run upstream-foo/upstream-foo

prefect deployment run upstream-bar/upstream-bar

prefect deployment run upstream-baz/upstream-baz

you'll see

downstream

get kicked off

MarcoM

09/05/2023, 7:52 AM

all clear thanks a lot @Nate, do you think there is a way to replicate this without automations in the Prefect Server solution?

Nate

09/05/2023, 2:14 PM

way to replicate this without automations in the Prefect Server solution

there's not a way to exact replicate this in OSS without cloud's automations, but there ways around that. off the top of my head the simplest way would be to write a wrapping parent flow that essentially just calls them all in order (e.g. wait for the upstream 10 and then call the downstream) using either the actual

flow

objects themselves or via

run_deployment

4 Views

Open in Slack

Previous Next