Hello Prefect folks! I'm searching for a pattern to manage dependencies of deployments. The 'hello world' way of calling a flow from another flow isn't a good fit as you will see by the components of my small data engineering pipeline: 1. Deployments A, B, C: DDL flow that creates tables on a DB. The flow is parametrized, the parameter is the filename of a DDL SQL script sent to a DB. So I have 3 deployments of same flow calling different scripts. 2. Deployment X: Data extracts that pull from tables built in deployments A, B and C. This is a single flow parametrized by the filename of several extract SQL scripts. I want to make sure Deployment X runs only after Deployments A, B, C have run each day. Currently I manage ordering by the cron schedule, however this isn't reliable if the agent isn't continuously running, or if a DDL job takes long time, etc. Can I have Deployment X explicitly wait on runs for other deployments (parametrized) run date?

This is a tricky one, I believe this will be handled by automations in the future cc @Will Raphaelson.

Hmm, so currently I believe you could satisfy this with simple automations created via UI if you were okay with the ddls running in parallel. You could set up automations to run X only after C only after B only after A. Because A, B, and C are of the same flow, youd want to listen for the flow run completions by tag. If you were set on the DDLs running in parallel, I believe you could create one automation that would satisfy this via the API by using the threshold - i.e wait for 3 flow runs with the DDL tag to enter completed before running X.

is there a way to check the state of a deployment run without actually calling the

run_deployment

method?

if previously_run_deployment(parameter1 = "2022-12-25").state.name = "Completed":
    # go ahead with this dependent flow here

I'm open to refactoring my DDL and extract jobs, I'm looking for a pattern that handles a data engineering setup that manages DDL jobs and various extracts, some of which have dependencies.

You can read the flow run and look at the state

Thanks I'll try to learn how to do that. From here its more of a data engineer's design question. I guess the pattern starts looking like: • create a dependency checking flow, it takes table names as parameter, and run date. ◦ dependency check sees if DDL flow(table = my_table, rundate = "2022-12-31") state == completed. ◦ then proceed else wait for timeout • dependency checking flow used as subflow in all DDL, extract flows • every new "SQL job" has three parts: ◦ sql file to send to DB ◦ list of table dependencies (put in deployment yaml) ◦ create deployment for it