Tim Enders
07/20/2022, 8:30 PMmap
operator coming? That is a big stumbling block for us adopting 2.0. Thanks!Kevin Kho
07/20/2022, 9:24 PMTim Enders
07/20/2022, 9:26 PMmap
for. to parallelize across a local Dask cluster. We have some flows that have to make a lot of API calls and they already take an hour or so in 5x parallelKevin Kho
07/20/2022, 9:28 PM[task_one(x) for x in items]
this will be parallel compared to the for loop because for loop makes it easy to block executionTim Enders
07/20/2022, 9:29 PMKevin Kho
07/20/2022, 9:31 PMfor item in items:
a = task_one(item)
b = task_two(a).result()
will wait for a to complete from the previous loop iteration i think because of the result call. Very easy to run into this