Hi Prefect Community, question about Scaling Task.map method: It doesn't seem like that when using the map method for task it will scale to multiple K8s pods. Is there any way to make it do so? The only work around now is to just use sub flows inside for loops instead of using task map method if we want horizontal task scaling.

Hi @Haotian Li — it sounds like you’ll want to use a Dask task runner with a dask-kubernetes cluster with autoscaling of workers.

Thanks @Zanie! An additional question, if I use a ConcurrentTaskRunner flow to call a DaskTaskRunner flow, does that mean only the Dask Flow will be submitted to Dask Cluster, not the super flow?

Yep!