Vadym Dytyniak

12/20/2022, 3:14 PM
Hi. My question is related to prefect-dask collection. Depends on the image I use for my DaskCluster Prefect submits tasks to DaskTaskRunner with different behavior. My task does manipilations with dask.dataframe and in one case dask transformation is distributed on the same cluster, but in other case task is running on one worker and other just waiting. Does anyone has an idea what should be different to see such different behaviour? Thanks.

Khuyen Tran

12/20/2022, 5:21 PM
@Andrew Huang Do you have any insights into this?

Vadym Dytyniak

12/21/2022, 9:22 AM
Thanks, didn't know about it. I also found what is difference between my 2 versions. It distribute the work in the case where I register PipInstall plugin. Sounds weird, but that is how it works.