https://prefect.io logo
Title
c

Chris L.

08/08/2022, 3:07 PM
Hello Prefecters. A question about Prefect 2.0 deployment. I have one
scheduler
flow that takes a
subflow_key
. This
subflow_key
is passed into a curried function that dynamically creates a new Prefect flow. There are about 30 (and growing) different subflows that can be created dynamically. My problem: I would like to have a single-flow many-deployments (each deployment is associated with different types of subflows) setup Constraints: Because subflows are generated dynamically using the curried function, I can't separate subflows into parent flows and run
prefect deployment build
for each flow. What I've tried: 30 different deployment yaml files for 1
scheduler
flow with 30 different combinations of
subflow_keys
and schedules. My question: Is there a DRYer way to achieve the same setup. Each deployment file is identical except for 2 lines (parameters and schedule)? What is Prefect engineering's current take on this single-flow many-deployments paradigm? Will this be achievable via a single Prefect CLI command in the future (maybe with arrays of parameters / schedule flags passed into
prefect deployment build
)?
j

Jeff Hale

08/08/2022, 3:24 PM
Good questions, Chris. Deployments are getting some improvements. We are planning to work on options for more efficient multi-flow deployment builds soon. I would keep an eye on the PRs and make sure that will meet your needs. I’ll also pass along this scenario. I don’t know of a DRYer way at the moment.
💯 1
I heard back from Chris and I think he’s spot on in that your situation sounds like something we should consider addressing within the scheduler / parametrization side of things: “it’s hard to schedule changing parameter values on a deployment”. I’ll open an issue on GitHub so we can track it.
@Marvin open “Improve changing parameter values on a deployment”