https://prefect.io logo
Title
i

itay livni

07/13/2020, 2:52 PM
Hi - Under what scenarios would a flow take longer to run locally with the DaskExecutor than the LocalExecutor? Is to many small tasks one of them?
c

Chris White

07/13/2020, 2:54 PM
Hi @itay livni - this is usually caused by two things: - having lots of small, fast tasks - exchanging large data between tasks Specifically, the slow down is typically caused by serialization overhead in sending data to / from the dask workers. We have identified a few ways of speeding this up under the hood, so those improvements should be coming out in the next week or so!
👍 1
i

itay livni

07/13/2020, 3:00 PM
Thanks 👍
@Chris White Is
0.12.5
faster? On one flow I have seen 10 to 30% improvement in runtime performance. ~18 sec to ~13.
:upvote: 4
c

Chris White

07/22/2020, 11:55 PM
yup it sure is! we simplified some of the information we were passing around, resulting in faster serialization times and a little less code
👏🏼 1
glad you noticed 😄
i

itay livni

07/22/2020, 11:57 PM
Kudos 💯