Thread
#prefect-community
    r

    Riley Hun

    1 year ago
    Hello everyone, For ephemeral Dask Cluster creation and execution of a Prefect workflow, what is the recommended approach -- Resource Managers or Dask Gateway? We are using Dask Gateway right now, but recently learned about Resource Managers so contemplating if it's necessary to switch over. If anyone has some insight into some advantages/disadvantages of both approaches, including cost-effectiveness, scalability, etc., please kindly advise.@Eric Jurotich
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Riley Hun, I haven’t used the
    Gateway Cluster
    but I read up on this and I think I understand it. If something I say sounds off, just let me know. My understanding is that
    Dask Gateway
    is a server for managing clusters. You can create a
    Prefect
    DaskExecutor
    by passing the
    GatewayCluster
    to the
    DaskExecutor
    . Similarly, you can use the
    Resource Manager
    to spin up a cluster (and you can use the
    Gateway Cluster
    for that temporary cluster. That said, I don’t think the question is Resource Manager or
    Dask Gateway
    because they can be used together. So the comparison here isn’t cost-effectiveness or scalability, but what the Prefect use-case is for both approaches. If you are manipulating
    Dask DataFrames
    or writing
    Dask
    specific code, it is better to use the
    Resource Manager
    because
    Prefect
    does not pass data between tasks well and the
    Dask
    Cluster will handle garbage collection this way. If you are writing non-
    Dask
    code and parallelizing over the
    Dask
    engine with
    Prefect's map,
    then passing the
    GatewayCluster
    to the
    DaskExecutor
    is better.
    r

    Riley Hun

    1 year ago
    Got it! That is a very clear response and explains the differences very well. Thanks @Kevin Kho
    e

    Eric Jurotich

    1 year ago
    Yeah, thats really good to know. Thanks!