Milton Tavares Neto

05/07/2021, 2:05 PM
Hello everyone, Would it be possible to setup a local dask cluster to run prefect tasks and a remote one to run dask collections? We've tried to do it but it seems that we can only use onde dask client for both at the same time. If it's not possible then the only solution to have both parallelisms would be the one described here: right? Thanks

Kevin Kho

05/07/2021, 2:06 PM
Hi @Milton Tavares Neto! Yes that’s right. Prefect sends task to workers for parallelism so the Dask operations end up being done on a worker level. You would need to follow the solution outlined there.

Milton Tavares Neto

05/07/2021, 2:29 PM
Great Kho. Thanks.