Chris L.
08/08/2022, 3:07 PMscheduler
flow that takes a subflow_key
. This subflow_key
is passed into a curried function that dynamically creates a new Prefect flow. There are about 30 (and growing) different subflows that can be created dynamically.
My problem: I would like to have a single-flow many-deployments (each deployment is associated with different types of subflows) setup
Constraints: Because subflows are generated dynamically using the curried function, I can't separate subflows into parent flows and run prefect deployment build
for each flow.
What I've tried: 30 different deployment yaml files for 1 scheduler
flow with 30 different combinations of subflow_keys
and schedules.
My question: Is there a DRYer way to achieve the same setup. Each deployment file is identical except for 2 lines (parameters and schedule)? What is Prefect engineering's current take on this single-flow many-deployments paradigm? Will this be achievable via a single Prefect CLI command in the future (maybe with arrays of parameters / schedule flags passed into prefect deployment build
)?Jeff Hale
08/08/2022, 3:24 PMMarvin
08/08/2022, 3:46 PM