Hello everyone, For ephemeral Dask Cluster creation and execution of a Prefect workflow, what is the recommended approach -- Resource Managers or Dask Gateway? We are using Dask Gateway right now, but recently learned about Resource Managers so contemplating if it's necessary to switch over. If anyone has some insight into some advantages/disadvantages of both approaches, including cost-effectiveness, scalability, etc., please kindly advise. @Eric Jurotich

Hi @Riley Hun, I haven’t used the

Gateway Cluster

but I read up on this and I think I understand it. If something I say sounds off, just let me know. My understanding is that

Dask Gateway

is a server for managing clusters. You can create a

Prefect

DaskExecutor

by passing the

GatewayCluster

to the

DaskExecutor

. Similarly, you can use the

Resource Manager

to spin up a cluster (and you can use the

Gateway Cluster

for that temporary cluster. That said, I don’t think the question is Resource Manager or

Dask Gateway

because they can be used together. So the comparison here isn’t cost-effectiveness or scalability, but what the Prefect use-case is for both approaches. If you are manipulating

Dask DataFrames

or writing

Dask

specific code, it is better to use the

Resource Manager

because

Prefect

does not pass data between tasks well and the

Dask

Cluster will handle garbage collection this way. If you are writing non-

Dask

code and parallelizing over the

Dask

engine with

Prefect's map,

then passing the

GatewayCluster

to the

DaskExecutor

is better.

Got it! That is a very clear response and explains the differences very well. Thanks @Kevin Kho

Yeah, thats really good to know. Thanks!