Paco Ibañez

05/04/2023, 7:43 PM
Hello, I have the following use case: I have a catalog of Prefect flows and I need to be able to define a DAG with a subset of them (as yaml/json), where each node represents a flow with its required input arguments etc. The DAG file is passed to an orchestrator flow that parses it and runs each flow in Prefect as a subflow. Is this a valid use case for Prefect? Is there any library/specifcation to define such a DAG?

Kyle McChesney

05/04/2023, 7:51 PM
I know you can define a flow of flows in a static sense. I am not aware of anyone who has done this dynamic generation of flows. At least in prefect 1, it would likely require you to generate the new flow, then register it, and the run it. I am not sure all of that would be possible to do in the context of a single flow run but maybe. Would be super cool if you get it figured out, #show-us-what-you-got would love to see it