We would like to utilize Dask worker plugins (specifically

PipInstall

https://distributed.dask.org/en/latest/plugins.html#distributed.diagnostics.plugin.PipInstall to install dependencies on our workers at worker creation time). We are using a temporary cluster model as described here https://docs.prefect.io/orchestration/flow_config/executors.html#using-a-temporary-cluster. Is it possible to somehow obtain a reference the `DaskExecutor`’s distributed

client

https://github.com/PrefectHQ/prefect/blob/master/src/prefect/executors/dask.py#L209 from within our Flow code (or Task code) so that we can call

register_worker_plugin

? I’ve browsed through the codebase but I’m not sure where in the Flow execution process that the `DaskExecutor`’s

client

is yielded and used?

Hi @Sean Harkins, did you come across this thread? https://prefect-community.slack.com/archives/CL09KU1K7/p1616680588405300

@Kevin Kho Yes. Actually, @Ryan Abernathey and I are collaborating on this project. My question is in response to @Zanie response. We are using an ephemeral cluster so we need some hook to access the executor’s client from within the flow after the cluster is started so that we can register the plugin.

Gotcha. Did you try the

get_client()

inside a Task inside the Flow to get the client?

@Kevin Kho 👍 I am able to access the

client

that way. I’ll try to confirm that

register_worker_plugin

works correctly and report back.

That’s awesome