Hey I might have missed something in the docs, so I’ll ask here:
I have a kubernetesAgent which runs all my tasks, and we want to make stuff parallel. is it ok to use the mapping features (here) with a
LocalExecutor
? or is there something else I need to do?
the docs here state that I need to a
DaskExecutor
for mapping to work…but I’m not sure which mapping, and I’m also not sure I understand why…
k
Kevin Kho
09/22/2021, 2:00 PM
Hey @Nadav Nuni, the LocalExecutor is sequential. You can map with it but it won’t be. parallelized. The LocalDaskExecutor is a local multiprocessing pool and will parallelize for you on one machine. You can use mapping with all executors
n
Nadav Nuni
09/23/2021, 6:24 AM
@Kevin Kho, great, thanks for explaining!…I guess the docs could specify this in a clearer way though…
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.