Hi, I'm pretty sure my task graph executing on dask is running breadth first. I notice all my level tasks completing before a new one starts. I also saw that since V 0.12.0 that depth first was available. I'm using Prefect Core with dask, can anyone point me to where I can specify the dask executor to use depth first? Thanks so much!
02/04/2021, 12:45 AM
Hi Sanjay - you can’t really force the dask scheduler to run tasks in a certain order; more often than not dask will run tasks depth first but there are many reasons that dask may decide against that execution plan. For example, if your tasks are really fast or if dask is only working with a limited amount of parallelism (there are other reasons as well)
You might also double check that you aren’t unintentionally “reducing” the levels of your task graph anywhere
02/04/2021, 1:07 AM
Thanks for the quick response Chris. I'll look into why we may be experiencing this