I’m struggling to get DFE to kick in for a mapped task, we are using the Dask executor and version 0.15. The flow is really simple List->A->B where A and B map.
Dask will strongly prefer DFE, but it can’t be forced either when it’s scheduler decides
j
Jacob Blanco
05/24/2022, 11:50 PM
Hey Anna, yes that’s what we are doing, for some reason it’s waiting for all runs of A to finish. That’s interesting Kevin, is there any way to predict when DFE will kick in? In this case we were trying to manage disk space in a flow that looks like this: Download file -> Upload file -> Delete file. Maybe relying on DFE in this context is an antipattern.
k
Kevin Kho
05/25/2022, 12:46 AM
Can you share your Flow structure? I think there might be something wrong. I think DFE should really kick in there. How many items also?
j
Jacob Blanco
05/25/2022, 1:34 AM
Thanks. 20 total. I'm not at my computer right now, will share later. Thanks.
a
Anna Geller
05/30/2022, 9:58 AM
Checking after holidays: LMK if this is still an open issue @Jacob Blanco