Heya! At the team I'm currently with, we're argui...
# ask-community
j
Heya! At the team I'm currently with, we're arguing about how should we implement dependency between "jobs" e.g. flows in Prefect. We have various flows currently scheduled, each flow is atomic and will run independently. For example 1. 01:00 - A - Data ingest from some external source 2. 01:00 - B - Data ingest from some internal source 3. 03:00 - C - dbt project's build run 4. 05:00 - D - Reverse etl from the data warehouse to S3 5. ... A and B have no dependency Run C only, if A and B has completed successfully Run D only, if A and B has completed successfully One way I've imagined this could be accomplished in Prefect is with a "master" flow that implements the dependency logic and talk with Prefect's API to schedule (now) appropriate flow runs. You'd only schedule the deployment of master flow. Colleague of mine insists that all these flows must become subflows or tasks in a monolith flow. (basically same thing as the former, but all code base is jammed into one massive flow). I'm sure there are other ways to accomplish this same thing, though what would be the best practice when it comes to Prefect? I tried searching for answers on google, but got no specific use cases like this described.
n
hey @jpuris - are you all on Prefect Cloud? this sounds like a great use case for the Events system
there's a lot of flexibility in how you configure deployment triggers, but from your description I'm essentially suggesting: • put a
DeploymentTrigger
on both C and D that
expect
a
prefect.flow-run.Completed
event (as mine does above) from
A
and
B
, where you can
match_related
on the
prefect.deployment.name: A
(and B), and optionally configure a time window,
within
which those completed events must occur
j
Indeed! We're on Prefect Cloud. I've never used the events system to be fair. This sounds like a near perfect way to solve this, if I've understood this correctly. Do you perhaps have a handy link to a blog or tutorial where this is explained in more detail? šŸ™
n
we're in the middle of a docs overhaul for events, so there's more to come soon on that (we're a little light on reference material for events right now šŸ˜… sorry) i'm pretty sure what I have above nearly exactly what you'll need (if i have time I'll PR this into the docs later today with some context on whats happening), but in the meantime I would suggest opening up (the
Raw
tab) a
prefect.flow-run.Completed
event in the Event feed that was emitted from A or B and it will be more apparent what's available to match on feel free to ask clarifiers here!
ā¤ļø 1
j
That would be awesome, @Nate! Thank you so much for suggesting this!