I want to compile a workflow DAG from a set of primitive tasks on the fly. The idea is that the steps can be arbitrarily re-ordered and the final result should be compared. So, like a combinatorial way of generating flows from a set of task nodes.
I'm wondering if anyone has already done this or if there is anything you'd consider as being important prior to jumping in.
I'm thinking that as along as Input/Output (not considering side effects) are all of the same type that this should be doable on the fly in reaction to an api request
05/18/2022, 8:18 PM
If you use Prefect 2.0 you don't need a DAG at all and tasks are determined dynamically at runtime