Tom Forbes
05/17/2021, 8:41 PMdf.apply
directly - is that what you would do? We’d rather use Prefect for this directly if possible. But I’m not clear on how Prefect works with Dask - would a mapping task like this be the way to go? Would this scale to a high number of tasks (millions?), or would you perhaps map over the dask partitions instead?Kevin Kho
05/17/2021, 8:57 PMTom Forbes
05/17/2021, 9:06 PMKevin Kho
05/17/2021, 9:17 PMdf.apply
. Prefect map
works best for non-Dask objects like mapping over a Python list.Tom Forbes
05/18/2021, 10:59 AM