https://prefect.io logo
Title
j

Jacob Blanco

05/23/2022, 7:27 AM
I’m struggling to get DFE to kick in for a mapped task, we are using the Dask executor and version 0.15. The flow is really simple List->A->B where A and B map.
a

Anna Geller

05/23/2022, 9:20 AM
Have you seen this blog post? https://medium.com/the-prefect-blog/map-faster-mapping-improvements-in-prefect-0-12-0-7cacc3f14e16 you can call taska.map(taskb.map(some_list))
k

Kevin Kho

05/23/2022, 2:35 PM
Dask will strongly prefer DFE, but it can’t be forced either when it’s scheduler decides
j

Jacob Blanco

05/24/2022, 11:50 PM
Hey Anna, yes that’s what we are doing, for some reason it’s waiting for all runs of A to finish. That’s interesting Kevin, is there any way to predict when DFE will kick in? In this case we were trying to manage disk space in a flow that looks like this: Download file -> Upload file -> Delete file. Maybe relying on DFE in this context is an antipattern.
k

Kevin Kho

05/25/2022, 12:46 AM
Can you share your Flow structure? I think there might be something wrong. I think DFE should really kick in there. How many items also?
j

Jacob Blanco

05/25/2022, 1:34 AM
Thanks. 20 total. I'm not at my computer right now, will share later. Thanks.
a

Anna Geller

05/30/2022, 9:58 AM
Checking after holidays: LMK if this is still an open issue @Jacob Blanco