Hello 👋 can anyone help clarify my understanding how LOOPing and Context works between workers, especially (if it makes a difference) when using a distributed Dask cluster. Presumably the new task that is created is not necessarily executed on the same worker? And if not, can I safely rely on the newly LOOP’d task having the latest context? Here I’m specifically interested to ensure task_loop_result will reliably have the latest value but I am also generally interested to better understand how Context etc. is managed between workers. Hope that makes sense - feel free to point me at anything if this already explained elsewhere! Thanks! Andy
đź‘‹ 1
c
Chris White
05/13/2020, 7:15 PM
Hi Andy! For looping specifically, each iteration of the loop will be run on the same worker (unless the worker crashes).
Context is never “shared” between task runs - new keys / values are introduced as the pipeline drills down from flow run -> individual task runs. If you are running your flows with a Prefect backend Prefect takes steps to ensure
task_loop_result
is always present and up-to-date for your looped task runs
a
Andy Waugh
05/13/2020, 7:59 PM
ah I see! That makes sense. Thanks for the swift and detailed response Chris!