Hi all, is there any way to disable tasks in the UI when I run a flow manually?
I’m migrating flows from azkaban to prefect, and this is smth azk supports. Would be nice if prefect could do the same thing.
The only way I can see rn to accomplish this would be to put case blocks in my code and add a parameter for each task to disable the task. This feels a bit overcomplicated though. Any help is welcome!
9 months ago
@brian that’s super interesting! Based on my limited Azkaban knowledge, I think Azkaban can do it because each task is a self-contained “job”, therefore Azkaban can temporarily disable the dependencies for a specific run allowing you to skip specific tasks.Prefect has a completely different compute model and because it provides a first-class support for data sharing between tasks (which Azkaban doesn’t have), you can’t just disable one task in the middle because the input data to the next task would be missing. While you could, as you suggested, build a conditional flow to skip specific tasks based on certain condition, it would require that you reregister your flow.The easiest solution I can think of is building a flow of flows. If there is some process that you want to sometimes run independently, and sometimes orchestrate it from a parent flow, then the flow of flows pattern may be beneficial. Here are some links that show this orchestration pattern:
• https://docs.prefect.io/core/idioms/flow-to-flow.html• https://www.prefect.io/blog/flow-of-flows-orchestrating-elt-with-prefect-and-dbt/• https://www.prefect.io/blog/orchestrating-elt-on-kubernetes-with-prefect-dbt-and-snowflake-part-2
9 months ago
Thanks @Anna Geller I’ll share all this info with my team