Couple questions: - Is a flow being evaluated even...
# prefect-community
v
Couple questions: • Is a flow being evaluated even if it’s not running? Airflow automatically evaluates a DAG every ~5 minutes, and I’m wondering if Prefect does something similar (I prefer it doesn’t) • I may be looking at this incorrectly, but it seems like when using
map
, each complete
map
task must complete before the next task starts. Using the example provided on the Prefect website (https://docs.prefect.io/core/examples/twitter_to_airtable.html), replies are received, all replies must be transformed before being loaded into the table. Is it possible to not wait for all of the replies to be transformed? So after replies are extracted…each one just goes through the transform and load steps without having to wait for the complete for other replies? THANKS!
c
Hi Viv! Great questions: - no, we view re-parsing the DAG as an anti-pattern and so the code is “frozen” via
cloudpickle
and never re-evaluated - you are 100% correct; we call this feature “Depth First Execution” and it is prioritized for the short-term. You can track this issue here: https://github.com/PrefectHQ/prefect/issues/2041
v
Awesome! Thanks Chris!
@Chris White silly question that I’m can’t seem to find again in the docs, if I register a new version of a Flow while there’s one that’s still running, does the running flow finish?
j
@Viv Ian yes it does - the new flow won’t destroy the old runs and allow them to finish
👍 1
m
@Leon Kozlowski ^ very relevant to our work!