Hey guys, Is it possible to trigger one flow to run another? If so, is there a prefect recommended way of doing it? I have a workflow where there are some heavyweight tasks that do not change much with time (related to parsing and data normalization). Tasks downstream from these are driven by research and regularly require re-running. As Flow seems to be atomic when it comes to scheduling in prefect, what is the best way for me to structure my workflow in prefect? Thanks in advance.
2 years ago
Hi @karteekaddanki, if I understand correctly, your root goal is to avoid re-running these expensive tasks? If so, prefect has a few mechanisms for caching tasks to avoid rerunning - you can then structure your whole workflow as a single flow and let prefect determine when a task needs to be rerun.
@Jim Crist-Harif This sounds like what I should be using although my intuition was to use something similar to what @Mark McDonald suggested. I'd need to figure out a systematic way of clearing the cached results (I haven't read the link in detail yet) for the case when the parsing does goes wrong.