Chris Hart
07/30/2019, 10:16 PMChris White
DaskExecutor
will certainly gain you parallelism of the mapped tasks at each level of mapping; I’m not 100% sure if this is all you’re asking about, so let me know if there was something more subtle here that I’m missing
- one caveat with mapping is that the results (outputs) of all the mapped tasks will be “reduced”, i.e., gathered into memory, so just be mindful of this when considering your resource requirementsChris Hart
07/31/2019, 4:07 PM