@brian that’s super interesting! Based on my limited Azkaban knowledge, I think Azkaban can do it because each task is a self-contained “job”, therefore Azkaban can temporarily disable the dependencies for a specific run allowing you to skip specific tasks.
Prefect has a completely different compute model and because it provides a first-class support for data sharing between tasks (which Azkaban doesn’t have), you can’t just disable one task in the middle because the input data to the next task would be missing. While you could, as you suggested, build a conditional flow to skip specific tasks based on certain condition, it would require that you reregister your flow.
The easiest solution I can think of is building a flow of flows. If there is some process that you want to sometimes run independently, and sometimes orchestrate it from a parent flow, then the flow of flows pattern may be beneficial. Here are some links that show this orchestration pattern:
•
https://docs.prefect.io/core/idioms/flow-to-flow.html
•
https://www.prefect.io/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/
•
https://www.prefect.io/blog/orchestrating-elt-on-kubernetes-with-prefect-dbt-and-snowflake-part-2