A Dask question. I frequently use

Client.register_worker_plugin

(https://docs.dask.org/en/latest/futures.html#distributed.Client.register_worker_plugin) with my Dask clusters to install custom pip packages (e.g. install from dev branch for debugging.) If I am using prefect with an existing dask scheduler, how can I call

client.register_worker_plugin

? I tried

executor = DaskExecutor(
    address=cluster.scheduler_address,
    client_kwargs={"security": cluster.security}
)
executor.client.register_worker_plugin

but it looks like the

client

attribute on

executor

is None.

Hi! I think you may need to try getting the client from within a task and registering the worker plugin there? You'd then just make this the first task of your flow.

Actually it's even easier! I can just create a separate

dask.distributed.Client

, independent from prefect, and call

register_worker_plugin

from that Client

👍 I presume that won't work if it's an ephemeral cluster? but glad it's working

the cluster has to be created outside of prefect for that to work