https://prefect.io logo
Title
v

Viv Ian

04/14/2020, 4:18 PM
Couple questions: • Is a flow being evaluated even if it’s not running? Airflow automatically evaluates a DAG every ~5 minutes, and I’m wondering if Prefect does something similar (I prefer it doesn’t) • I may be looking at this incorrectly, but it seems like when using
map
, each complete
map
task must complete before the next task starts. Using the example provided on the Prefect website (https://docs.prefect.io/core/examples/twitter_to_airtable.html), replies are received, all replies must be transformed before being loaded into the table. Is it possible to not wait for all of the replies to be transformed? So after replies are extracted…each one just goes through the transform and load steps without having to wait for the complete for other replies? THANKS!
c

Chris White

04/14/2020, 4:21 PM
Hi Viv! Great questions: - no, we view re-parsing the DAG as an anti-pattern and so the code is “frozen” via
cloudpickle
and never re-evaluated - you are 100% correct; we call this feature “Depth First Execution” and it is prioritized for the short-term. You can track this issue here: https://github.com/PrefectHQ/prefect/issues/2041
v

Viv Ian

04/14/2020, 4:52 PM
Awesome! Thanks Chris!
@Chris White silly question that I’m can’t seem to find again in the docs, if I register a new version of a Flow while there’s one that’s still running, does the running flow finish?
j

Jeremiah

04/14/2020, 5:01 PM
@Viv Ian yes it does - the new flow won’t destroy the old runs and allow them to finish
👍 1
m

Mitchell Bregman

04/14/2020, 5:40 PM
@Leon Kozlowski ^ very relevant to our work!